Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I was wondering if a Perl module exists that can:
A) visit given URLS, and
B) save the contents of these pages
I have been looking at CPAN, but maybe not hard enough since I haven't found anything.
Any words of wisdom?

Edited 2003-03-19 by mirod: changed the title.

2006-10-25 Retitled by planetscape, as per Monastery guidelines

( keep:3 edit:18 reap:0 )

Original title: 'a beginner'

Replies are listed 'Best First'.
Re: Perl module for downloading web pages
by Aristotle (Chancellor) on Mar 17, 2003 at 03:11 UTC
    Have a look at the LWP family of modules. Also of note, depending on what you want, may be the modules found in the Net:: namespace.

    Makeshifts last the longest.

[reply]
Re: Perl module for downloading web pages
by pg (Canon) on Mar 17, 2003 at 03:12 UTC
    Maybe you want start with LWP::UserAgent, and HTTP::Request, both are core modules coming with standard Perl distribution.

    If you are familiar with http 1.1 spec, and socket programming, and are interested in details, you can also play with IO::Socket::INET.

    Example: 
    use LWP::UserAgent;
use HTTP::Request;

use strict;

my $agent = new LWP::UserAgent();
my $req = new HTTP::Request(GET => "http://www.google.com");
my $res = $agent->request($req);
print $res->content if ($res->is_success)

    [download]
[reply]
[d/l]
      ...and LWP::Simple. 
          # poor man's wget:
    % perl -MLWP::Simple -e "getprint(shift)" http://www.perl.org/ > p
+erl.org.index.html

      [download]

      jdporter
      The 6th Rule of Perl Club is -- There is no Rule #6.

[reply]
[d/l]
Re: Perl module for downloading web pages
by silent11 (Vicar) on Mar 17, 2003 at 03:44 UTC
[reply]
Re: Perl module for downloading web pages
by dash2 (Hermit) on Mar 17, 2003 at 09:47 UTC
    The standard modules for this stuff are in LWP, but if you want a higher level of abstraction for wandering round the web, you could try WWW::Robot. It's good at following links.

    And in general, you should use WWW::RobotUA from the LWP distribution if you're going to be doing a lot of automatic screen scraping.

    andramoiennepemousapolutropon

[reply]
Re: Perl module for downloading web pages
by MrYoya (Monk) on Mar 17, 2003 at 04:06 UTC
    It sounds like LWP would be very useful to look at. I would like to add that perldoc lwpcook contains some more documentation and code examples about LWP. There's even an entire O'Reilly book dedicated to it called "Perl & LWP". It's very easy to use, in fact LWP was what brought me to perl in the first place.
[reply]
[d/l]
Re: Perl module for downloading web pages
by Steve_p (Priest) on Mar 17, 2003 at 15:01 UTC
    For a more detailed work on Perl web programming, you can also check out Web Client Programming with Perl. With the book free on the web now, you can't beat the price.
[reply]

Back to Seekers of Perl Wisdom