in reply to (OT) Robots disallow

You can't. There is no possible way for a server to tell the difference between a robot and a human user. The Robots_exclusion_standard is purely voluntary for robot scripters, and you have already followed the protocol.

Replies are listed 'Best First'.
Re^2: (OT) Robots disallow
by jhourcle (Prior) on Apr 01, 2009 at 18:23 UTC

    If there's a specific signature of the bot that you're having problems with (a specific domain, or an identifiable user-agent), look into configuring your webserver to send them alternative content.

    Back when it was easy to identify e-mail harvesters, I'd send them to a CGI that slowly feeds them bogus e-mail addresses and the abuse address from the netblock they're coming from. These days, most harvesters are coming from botnets, so the abuse one isn't so useful. (and yes, it _is_ a Perl script)