in reply to Passing timeout params through WWW::RobotRules
What is fetching your $robots_txt file? As far as I understand, WWW::RobotRules does not fetch files itself, but rather expects you to use some user agent like LWP::Simple or LWP::UseAgent for that... And with the latter, you could then just call the timeout method on the user agent object, e.g.
use LWP::UserAgent; my $ua = LWP::UserAgent->new; $ua->timeout(2); ...
or, when you're using LWP::Simple, you have to import $ua to get access to its internal LWP::UserAgent object:
use LWP::Simple qw(get $ua); $ua->timeout(2); ...
Or better yet, simply use LWP::RobotUA in the first place, which should provide the same timeout method, as it's a sub-class of LWP::UserAgent. Something like this (if I'm reading the docs correctly — i.e. untested):
use LWP::RobotUA; use WWW::RobotRules; my $ua = LWP::RobotUA->new(...); my $rules = WWW::RobotRules->new(...); $ua->rules($rules); # optional - defaults used otherwise $ua->timeout(2); my $response = $ua->get(...); ...
|
|---|