in reply to perl regex or module that identifies bots/crawlers
RewriteEngine On RewriteCond %{HTTP_USER_AGENT} ^BadBot.* RewriteRule .* - [F]
Another solution could be to use a rewrite map that contains all the bad bots you know:
This does require the use of a dbm map, since the USER_AGENT can contain spaces. It also requires exact matches, so you need to keep your map up to date. It should be very fast though.RewriteMap robotmap dbm:/path/to/file/map.db RewriteCond %{HTTP_USER_AGENT} !="" RewriteCond ${robotmap:%{HTTP_USER_AGENT}|NOT-FOUND} != NOT-FOUND RewriteRule .* - [F]
See also mod_rewrite manual.
|
|---|