adamsj has asked for the wisdom of the Perl Monks concerning the following question: 

use strict;
use LWP;

my $url = 'ftp://path/to/a/greatbigfile';

my $agent = LWP::UserAgent->new;
my $request = HTTP::Request->new(GET => $url);
$agent->proxy('ftp' => 'http://our.proxy.server:8080');

my $response = $agent->request($request);
$response->is_success or die "$url: ", $response->message,"\n";

open(MYOUT, ">/home/greatbigfile.") or die "No open? $!";
print MYOUT $response->content;
close(MYOUT);

[download]

Nothing wrong with the code per se--it works fine on itty bitty files--but it fails like this: 

ftp://path/to/a/greatbigfile: Out of memory during "large" request for
+ 33558528 bytes, total sbrk() is 37230796 bytes

[download]

on a file of nearly 40 meg. Is there any way to pull the file in chunks? I don't find it in the documentation.

adamsj

They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Replies are listed 'Best First'.
Re: LWP file transfer running out ot memory on a whopping big file
by gav^ (Curate) on Feb 07, 2002 at 13:56 UTC
    If you type perldoc lwpcook or go here you will find loads LWP examples including one about handling large documents.

    A quick copy paste is: 

      use LWP::UserAgent;
  $ua = LWP::UserAgent->new;
  my $req = HTTP::Request->new(GET =>
                'http://www.linpro.no/lwp/libwww-perl-5.46.tar.gz');
  $res = $ua->request($req, "libwww-perl.tar.gz");
  if ($res->is_success) {
     print "ok\n";
  }
  else {
     print $res->status_line, "\n";
  }

    [download]

    gav^

[reply]
[d/l]
      It's workin' like a charm--thanks! 

      for ($i = 0; $i < 1000; $++) {
     print "I _will_ remember to read lwpcook as part of the lwp docum
+entation.\n";
}

      [download]

      adamsj

      They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

[reply]
[d/l]
        for ($i = 0; $i < 1000; $++)

        Whoopsie, you forgot an 'i'. 

        print "I wll remember my i's\n" for ( 0..999);

        [download]
        Couldn't resist ;)

        blyman
        setenv EXINIT 'set noai ts=2'

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom