in reply to Re: Using URI::URL
in thread Using URI::URL

does www::mechanize follow the rules set by lwp::robotua? I know that before ifi tried to "get" something that a robots.txt file didnt allow me too, my get came up empty (a good thing). Does www:mechanize allow me to do the same thing? I have been to CPAN and looked at it, but didnt see anything about obeying rules. Thanks

Replies are listed 'Best First'.
WWW::Mechanize::Polite ?
by jeffa (Bishop) on Feb 22, 2004 at 02:58 UTC
    Kudos to you for wanting polite bots. The problem with getting LWP::RobotUA to play nice with WWW::Mechanize is that they both are subclasses of LWP::UserAgent. By itself, WWW::Mechanize does not consult the /robots.txt file, but you can instead use WWW::RobotRules. Here is a working example that tries to grab two files from my server:
    Read more... (856 Bytes)
    There might be a better way though ... ahh, how about "WWW::Mechanize::Polite"?
    Read more... (1364 Bytes)
    And if i didn't just reinvent a wheel, you might be seeing this on the CPAN. ;)

    jeffa

    L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)
[reply]
[d/l]
[select]
      thanks jeffa, you rock!
[reply]

In Section Seekers of Perl Wisdom