Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

My question touches closely on this recent thread. I am curious if anyone has been able to do FTP uploads (PUTs) of very large files using LWP without consuming a lot of memory. I tried out something like this code while watching a system monitor (top/gtop/etc) and it ate up lots of memory:

use LWP::UserAgent;
my $url = 'ftp://anonymous:anon@ftp.someserver.com/pub/100mb.file';
my $ua = LWP::UserAgent->new();
$ua->agent("$0/.01 ".$ua->agent);
open (READER, "100mb.file") or die "Can't read: $!\n";
my $req = HTTP::Request->new('PUT', "$url", undef, <READER>);
my $result = $ua->request($req);

[download]

Having never really needed to FTP large files, I never realized how much memory this could eat up. Now I'm worried that some old scripts we have laying around need a bit of reworking in case anyone every tries them with big files. For the sake of some other folks around who aren't familiar with Net::FTP, I'd like to continue to do this using LWP if I can. Anyone know how to reduce the amount of memory needed for something like this?

Replies are listed 'Best First'.
Re: Reducing LWP Buffering
by Roger (Parson) on Dec 05, 2003 at 00:04 UTC
    What's wrong with the Net::FTP module? It's extremely easy to use, even LWP uses Net::FTP. I have written a small example using the Net::FTP module to do the samething. Net::FTP isn't exactly rocket science and should be easily mastered. ;-)

    #!/usr/bin/perl -w
use strict;
use Net::FTP;

my $ftp = Net::FTP->new("ftp.someserver.com")
  or die "Cannot connect to ftp.someserver.com: $@";

$ftp->login("anonymous",'anon')
  or die "Cannot login ", $ftp->message;

$ftp->cwd("/pub")
  or die "Cannot change working directory ", $ftp->message;

$ftp->put("100mb.file")
  or die "put failed ", $ftp->message;

$ftp->quit;

    [download]
[reply]
[d/l]
Re: Reducing LWP Buffering
by Anonymous Monk on Dec 05, 2003 at 00:06 UTC
    Didn't you learn anything from that recent thread? You're reading the entire file into memory. If you can't suffer the memory requirement, don't use LWP::UserAgent for FTP, use what LWP uses.
[reply]
Re: Reducing LWP Buffering
by Anonymous Monk on Dec 05, 2003 at 12:56 UTC
    Yeah, yeah, I've heard the answer before: use Net::FTP. But that's not what Anonymous really asked. With perl there's almost always more than one way to do it, but I'm curious to see if can be done this way--with LWP. Is this simply a limitation of the LWP module? Is there no way to tell LWP ow to do buffering? FWIW, I ran into a similar situation once while working on an embedded platform that made adding modules very difficult. It's nice to be working on PC platforms again, where adding modules is easy, but I will never again tell someone who for some reason resists using some module to use it anyway. I'll be watching this thread to see if the problem can be solved, methinks....
[reply]

      I think that should be possible by simply giving the HTTP::Request a code reference as value for the content key, and that code reference then supplies the content. I don't know why this wasn't mentioned yet, but I'm too lazy to check it myself in the documentation.

      Update: Quoth the LWP::UserAgent documentation:

      $ua->request( $request, $content_cb )

      [download]

      ...

      The request methods described above; get(), head(), post() and mirror(), will all dispatch the request they build via this method. They are convenience methods that simply hides the creation of the request object for you.

      The $content_file, $content_cb and $read_size_hint all correspond to options described with the get() method above.

      You are allowed to use a CODE reference as content in the request object passed in. The content function should return the content when called. The content can be returned in chunks. The content function will be invoked repeatedly until it return an empty string to signal that there is no more content.

      I know it was there somewhere.

      perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ;    # The  
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new   #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' #  web

      [download]
[reply]
[d/l]
[select]
        I don't know why this wasn't mentioned yet

        Probably the same reason the original seeker didn't look at the docs.

[reply]
        Having never tried that, I'm wondering what the code might look like. Would you need to do something like:

        open (READER, "$file") or die "Can't read: $!\n"; 
my $req = HTTP::Request->new('PUT', "$url", undef, \&read_file_in_chun
+ks(\*READER));


sub read_file_in_chunks {
     my ($fh_ref) = @_;
     my $content;
     read ($fh_ref, $content, 51200);
     return $content;
}

        [download]

        Hmmm..actually, I just tried that and it seems only the first 51200 bytes get uploaded. What did I miss?
[reply]
[d/l]
      but I'm curious to see if can be done this way--with LWP
      Can you see in the documentation?
[reply]

Back to Seekers of Perl Wisdom