LWP file size

grasshopper!!! has asked for the wisdom of the Perl Monks concerning the following question:

I have tried to download the 8mb index.all file from the Gutenburg Project website with LWP::UserAgent but it only gets around 5mb. With wget I gets the full file, is there some limit in LWP? I have tried LWP::Agent many times and it gets between 4mb-5.2mb.

my $url = 'https://www.gutenberg.org/dirs/GUTINDEX.ALL';
     my $file = '/home/philip/APPLICATIONS/GUTINDEX.ALL';
     say $url;
     my $browser = LWP::UserAgent->new();
     my $response = $browser->get($url);
     open my $fp,">",$file || die "Error saving book $!\n";
     print $fp $response->content();
     close $fp;
[download]

Comment on LWP file size Download Code

Replies are listed 'Best First'.
Re: LWP file size by hippo (Archbishop) on Jun 12, 2025 at 11:30 UTC
If you want to download a large file (or any file, really), use `mirror` in preference to `get`. Note also that `open my $fp,">",$file \|\| die "Error saving book $!\n";` isn't going to report the errors in the way that you intend because the comma has a lower priority than the `\|\|` operator. Use `or` for control flow instead. Of course if you used `mirror` then you don't need to worry about the mechanics of writing to the file in the first place :-) 🦛	[reply] [d/l] [select]
Re: LWP file size by Corion (Patriarch) on Jun 12, 2025 at 10:18 UTC
Have you checked that the size of the content before writing it to the file matches the size that the web server tells you? `say length $response->content(); say $response->header("Content-Length");` [download] Also have a look at the ->add_handler method to add a progress callback so you see where in the HTTP processing you are.	[reply] [d/l]