Re: Downloading html content

pupun,

You've probably got all you need. However, if not, the following bit of code might help.


#!/usr/bin/perl

use warnings;
use strict;


use LWP::UserAgent;
use HTTP::Request::Common;


my @web_addresses = ("http://www.web1.com/page",
"http://www.web2.co.uk/page.htm");


my $count = 0;

foreach (@web_addresses) {

  $count++;

  my $ua = LWP::UserAgent->new;
  $ua->get("$_", ':content_file' => "$count");

}
[download]

I expect that the code may work without use HTTP::Request::Common;. Not sure, you will have to try that out if you wish to know.

Comment on Re: Downloading html content Select or Download Code

Replies are listed 'Best First'.

Re^2: Downloading html content
by pupun (Initiate) on Jan 03, 2008 at 09:37 UTC

Thanks for your reply. Let me put it more clearly. actually, it's like this; there is a particular secured client site in that particular site through the script we have to give the user id and password and once we enter the site the script will access a partcular link and from which the script has to download data and then save it in a network drive. OS- windows 2000

[reply]

Re^3: Downloading html content

by Jenda (Abbot) on Jan 04, 2008 at 01:43 UTC

Let me put it more clearly. If you looked at the modules suggested by other monks you would find out that the task is fairly easy even if you have to login. You can either use WWW::Mechanize to navigate to the login page, fill in the form, submit it and then navigate to that "particular link" and save content. Or you can use LWP::UserAgent to send the login information to the same URL the login form would send it making sure the cookies sent with the response are remembered by the LWP::UserAgent object and then download that particular link.

But I admit it would require that you look at the documentation of those modules and try to do something yourself.

Show us you tried and come back if you have problems, but don't expect to get a full script doing exactly what you ask for. This is not a free script service.

Jenda
Support Denmark!
Defend the free world!

[reply]

A reply falls below the community's threshold of quality. You may see it by logging in.