in reply to Re: perl regex or module that identifies bots/crawlers
in thread perl regex or module that identifies bots/crawlers

I agree that the real Google and Yahoo, and other big ones, will certainly honor robots.txt. If bots under their names invade a server that may only indicate that these are popular fake names for rogue bots. It would make sense to look like a legit bot instead of, for instance, a browser.

That said, it is certainly a good idea to check if robots.txt is working as it should.

Anno