LWP::UserAgent and sending the HTTP

mellin has asked for the wisdom of the Perl Monks concerning the following question:

I created a small script to download all the images from any given HTML page. It works otherwise just fine, but some sites are using redirection to their index page unless the user arrives onto this page from their site (server reading the HTTP_REFERER).

I took this into consideration in my script after reading a bit more about this from Google. Below is a snippet of the script that fails and the error, too

$http = LWP::UserAgent -> new;
$http->timeout(10);
$http->agent("Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)");
$http->header('Referer' => "$referer") if defined $referer; # send HTT
+P_REFERER if requested
$content = $http->get($src);
[download]

www:~/Documents/testStuff/getImagesFromWww mellimik$ perl getImagesFro
+mWww.pl http://localhost/


total of 1 images

Can't locate object method "header" via package "LWP::UserAgent" at ge
+tImagesFromWww.pl line 86.
www:~/Documents/testStuff/getImagesFromWww mellimik$
[download]

Comment on LWP::UserAgent and sending the HTTP_REFERER Select or Download Code

Replies are listed 'Best First'.
Re: LWP::UserAgent and sending the HTTP_REFERER by rpanman (Scribe) on Jul 21, 2007 at 14:42 UTC
The version of LWP::UserAgent that I have does not have a method called "header", I imagine that yours is the same; hence the error. Looking at the docs it seems like the following may work for you: `$ua->default_header( 'Referer' => $referer ) if defined $referer;` [download]	[reply] [d/l]
Re: LWP::UserAgent and sending the HTTP_REFERER by dsheroh (Monsignor) on Jul 21, 2007 at 15:00 UTC
If you're going to use raw LWP, you need to create a separate HTTP::Request object and set its `->referer` property, then pass that to the LWP::UserAgent: `my $req = HTTP::Request->new(); $req->method('GET'); $req->url($uri); $req->referer($referer) if $referer; $content = $http->request($req);` [download] Or you may want to take a look at WWW::Mechanize if you want an all-in-one package. Its docs include referer manipulation in one of the examples, so it would appear to be capable of this.	[reply] [d/l] [select]