Re: Re: Re: Downloading image files using LWP

You are using LWP::UserAgent::mirror() which does the local caching. That checks for the local file, uses its timestamp in a If-Modified-Since header, and does a conditional GET.

Since you want to force the file to be downloaded, either don't use mirror, or delete the local file before you call it.

The UserAgent request method takes a filename as the second parameter. It will create (or overwrite) the file with the downloaded contents. You should check that the download succeed and returned the expected number of bytes.

Comment on Re: Re: Re: Downloading image files using LWP Select or Download Code

Replies are listed 'Best First'.
Re: Re: Re: Re: Downloading image files using LWP by gnangia (Scribe) on Nov 06, 2002 at 18:37 UTC
That was it. I ran a sniffer trace and found that the request was sending in the header "IF-Modified-Since" which was causing google.com to send a reply with html code 304 (Not Modified). So I modified my last subroutine loop where I request the image url as follows - `my $img_url = $attr->{src}; my $remote_name =URI->new_abs($img_url,$parser->{base}); my $local_name=$remote_name->host . $remote_name->path; mkpath(dirname($local_name),0,0711); print "Getting imagefile: $img_url\n"; $request = HTTP::Request->new(GET => $remote_name); my $response = $browser->request($request, $local_name); print STDERR "YYY-$local_name: ",$response->status_line,` [download] It works now. Thanks to everyone for their input.	[reply] [d/l]

Replies are listed 'Best First'.

Re: Re: Re: Re: Downloading image files using LWP
by gnangia (Scribe) on Nov 06, 2002 at 18:37 UTC

my $img_url = $attr->{src};                             
my $remote_name =URI->new_abs($img_url,$parser->{base});
                                                        
my $local_name=$remote_name->host . $remote_name->path;
mkpath(dirname($local_name),0,0711);                   
                                                        
print "Getting imagefile: $img_url\n";                  
$request  = HTTP::Request->new(GET => $remote_name);    
my $response = $browser->request($request, $local_name);
                                                        
print STDERR "YYY-$local_name: ",$response->status_line,
[download]

[reply]
[d/l]