perlpreben has asked for the wisdom of the Perl Monks concerning the following question:
require WWW::RobotRules; my $user_agent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)"; my $robotsrules = new WWW::RobotRules '$user_agent'; ..... $robotsrules->parse($url, $robots_txt); if($robotsrules->allowed($url)){ }
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: Passing timeout params through WWW::RobotRules
by almut (Canon) on Apr 17, 2010 at 15:49 UTC | |
|
Re: Passing timeout params through WWW::RobotRules
by trwww (Priest) on Apr 17, 2010 at 16:12 UTC | |
by perlpreben (Beadle) on Apr 17, 2010 at 18:03 UTC |