Jump to content

How to block unknown robots with mod_rewrite?


longtone

Recommended Posts

In cPanel there is a bot:

 

"Unknown robot (identified by 'bot*')"

 

and it's using 30% of of the bandwidth

 

I used mod_rewrite to block a number of known bad bots, and added the line:

RewriteCond %{HTTP_USER_AGENT} ^bot*

 

The bandwidth used by bot* did decrease (from about 40%), but it didn't stop

 

I'm not sure if I'm doing this right, whether * is a wildcard, and whether I should be putting bot.* instead of bot*, or if it's an actual asterisk and I should be escaping it.

 

Or maybe it's a whole range of different bots, all starting with 'bot'

 

Or is it possible (or desirable) to block all robots except Google, yahoo and MSN?

 

Any ideas?

 

You would do something like this:

RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]

 

The NC makes it case insensitive, the F means it'll give a 403 Forbidden response code and the L means the mod_rewrite chain stops on a successful match.

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.