Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Out of memory while using LWP::UserAgent

by ezekiel (Pilgrim)
on Jan 04, 2006 at 21:12 UTC ( [id://521020]=perlquestion: print w/replies, xml ) Need Help??

ezekiel has asked for the wisdom of the Perl Monks concerning the following question:

I am using LWP::UserAgent to retreive files from a FTP site. Everything appears to work fine until I attempt to get a file that is supposedly 65M in size. At this point the script dies with error:

"Out of memory during "large" request for 134221824 bytes, total sbrk() is 271669258 bytes at .."

The code I am using is below. I know the physical memory is available. Any thoughts on how to get around this problem? Thanks in advance.

my $ua = LWP::UserAgent->new(); my $response = $ua->get("$url"); if ($response->is_success) { my $content = $response->content; open (OUTPUT, ">out.txt") or die "Cannot open the output file: $!" +; print OUTPUT "$content"; close (OUTPUT) or die "Cannot close the output file: $!"; } else { die $response->status_line; }

Replies are listed 'Best First'.
Re: Out of memory while using LWP::UserAgent
by Fletch (Bishop) on Jan 04, 2006 at 21:28 UTC

    See the section "Large Documents" in perldoc lwpcook.

Re: Out of memory while using LWP::UserAgent
by ambrus (Abbot) on Jan 05, 2006 at 12:22 UTC

    Search for content_cb and content_file in perldoc LWP::UserAgent. This shows how you can write the data to a file or process it with a callback instead of writing to a perl scalar.

Re: Out of memory while using LWP::UserAgent
by kwaping (Priest) on Jan 04, 2006 at 22:24 UTC
    Maybe Net::FTP would better suit your needs?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://521020]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others romping around the Monastery: (6)
As of 2024-04-24 22:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found