in reply to RobotUA not working

Definitely not directly related but neither yahoo nor google or alexa or ... like being spidered. They have some very serious anti DOS/spider firewalling happening. To spider sites like these is a challenge, although it can of course be done. For a start don't admit to being a robot, pretend to be IE or if you want to pretend to be a robot make yourself googlebot or one of the well known spiders that people welcome.

I presume you are aware of robots.txt and what it does.

cheers

tachyon

Replies are listed 'Best First'.
RobotUA not working
by mkurtis (Scribe) on Mar 07, 2004 at 19:00 UTC
    thanks Tachyon, however since the whole point of using robotUa was to follow the rules, i could have used mechanize and pretended to be a browser, but they can block you in thier htaccess file for that. Anyway googlebot isnt written in PERl and i dont think they would buy it was google bot anyway, besides some sites let googlebot index less than other bots. Thanks for the post though, if my RobotUA delay feature doesnt stop taking minutes, i might just resort to mechanize and your idea.
    Thanks
[reply]

In Section Seekers of Perl Wisdom