Re: Using URI::URL

Wrong question. You are trying to pull links from a webpage. You most likely instead should be using WWW:Mechanize:

use strict;
use warnings;
use Data::Dumper;
use WWW::Mechanize;

my $a = WWW::Mechanize->new();
$a->get( 'http://some.site.com.' );

print $_->url,"\n" for @{ $a->links };
[download]

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

Comment on Re: Using URI::URL Download Code

Replies are listed 'Best First'.
Re: Re: Using URI::URL by Anonymous Monk on Feb 21, 2004 at 17:53 UTC
does www::mechanize follow the rules set by lwp::robotua? I know that before ifi tried to "get" something that a robots.txt file didnt allow me too, my get came up empty (a good thing). Does www:mechanize allow me to do the same thing? I have been to CPAN and looked at it, but didnt see anything about obeying rules. Thanks	[reply]
WWW::Mechanize::Polite ? by jeffa (Bishop) on Feb 22, 2004 at 02:58 UTC
Kudos to you for wanting polite bots. The problem with getting LWP::RobotUA to play nice with WWW::Mechanize is that they both are subclasses of LWP::UserAgent. By itself, WWW::Mechanize does not consult the `/robots.txt` file, but you can instead use WWW::RobotRules. Here is a working example that tries to grab two files from my server: Read more... (856 Bytes) There might be a better way though ... ahh, how about "WWW::Mechanize::Polite"? Read more... (1364 Bytes) And if i didn't just reinvent a wheel, you might be seeing this on the CPAN. ;) jeffa L-LL-L--L-LL-L--L-LL-L-- -R--R-RR-R--R-RR-R--R-RR B--B--B--B--B--B--B--B-- H---H---H---H---H---H--- (the triplet paradiddle with high-hat)	[reply] [d/l] [select]
Re: WWW::Mechanize::Polite ? by mkurtis (Scribe) on Feb 22, 2004 at 03:14 UTC
thanks jeffa, you rock!	[reply]