cormanaz has asked for the wisdom of the Perl Monks concerning the following question:

Good day monks. I am using WWW:Mechanize to download zip files from a web page. Download is initiated by clicking a form button. What I'm currently using is:
use WWW::Mechanize; my $mech = WWW::Mechanize->new(); $mech->get($url); if ($mech->success()) { $mech->form_number(1); my $response = $mech->click_button(name => 'begin'); open(OUT,">save.zip") or die "Can't open outfile: $!\n"; binmode OUT; print OUT $response->content(); close OUT; }
But when the file is large or I have a lot of other things running on my machine, the $response can cause memory shortages. I'm wondering if there is a way to stream the response straight to a file, i.e. like you can do with the get method using
$mech->get($url, ":content_file" => "filename.here");
only in response to click_button

Thanks...Steve

Replies are listed 'Best First'.
Re: WWW:Mechanize click_button stream to file?
by jettero (Monsignor) on Feb 12, 2007 at 14:43 UTC
    Afaik, WWW::Mechanize is in fact an LWP::UserAgent object (i.e., inheritance). So your get should work pretty much just like you typed it.

    Update: On further consideration, I see that I misunderstood the question. Sorry about that. I think the answer is actually: no.

    You'd have to locate the content by hand using the find URL functions and do your the way you want... I have a semi-mech solution in the readmore below. Hopefully there's a better way.

    -Paul

      But the problem is that the download is initaited by a button click, not a get. Steve
Re: WWW:Mechanize click_button stream to file?
by lorn (Monk) on Feb 12, 2007 at 19:24 UTC
      Same problem
Re: WWW:Mechanize click_button stream to file?
by Anonymous Monk on Feb 13, 2007 at 00:57 UTC
    UTSL
    use WWW::Mechanize; my $mech = WWW::Mechanize->new(); $mech->get('http://search.cpan.org/'); $mech->set_fields( query => 'ACME' ); click_button( $mech, value => 'CPAN Search', ":content_file" => "filename.here" ); sub click_button { my $self = shift; my %args = @_; for ( keys %args ) { if ( !/^(number|name|value|input|x|y)$/ ) { $self->warn( qq{Unknown click_button parameter "$_"} ); } } for ($args{x}, $args{y}) { $_ = 1 unless defined; } my $form = $self->{form}; my $request; if ( $args{name} ) { $request = $form->click( $args{name}, $args{x}, $args{y} ); } elsif ( $args{number} ) { my $input = $form->find_input( undef, 'submit', $args{number} +); $request = $input->click( $form, $args{x}, $args{y} ); } elsif ( $args{input} ) { $request = $args{input}->click( $form, $args{x}, $args{y} ); } elsif ( $args{value} ) { my $i = 1; while ( my $input = $form->find_input(undef, 'submit', $i) ) { if ( $args{value} && ($args{value} eq $input->value) ) { $request = $input->click( $form, $args{x}, $args{y} ); last; } $i++; } # while } # $args{value} return $self->request( $request, grep defined, @args{qw[:content_f +ile :content_cb :read_size_hint ]}); }
    luke master :)
      W::M is basically LWP::UserAgent. so you can also refer to perldoc lwpcook the 'large documents' section.
      LARGE DOCUMENTS
          If the document you want to fetch is too large to be kept in memory,
          then you have two alternatives. You can instruct the library to write
          the document content to a file (second $ua->request() argument is a file
          name):
      
            use LWP::UserAgent;
            $ua = LWP::UserAgent->new;
      
            my $req = HTTP::Request->new(GET =>
                          'http://www.linpro.no/lwp/libwww-perl-5.46.tar.gz');
            $res = $ua->request($req, "libwww-perl.tar.gz");
            if ($res->is_success) {
               print "ok\n";
            }
            else {
               print $res->status_line, "\n";
            }
      
          Or you can process the document as it arrives (second $ua->request()
          argument is a code reference):
      
            use LWP::UserAgent;
            $ua = LWP::UserAgent->new;
            $URL = 'ftp://ftp.unit.no/pub/rfc/rfc-index.txt';
      
            my $expected_length;
            my $bytes_received = 0;
            my $res =
               $ua->request(HTTP::Request->new(GET => $URL),
                         sub {
                             my($chunk, $res) = @_;
                             $bytes_received += length($chunk);
                             unless (defined $expected_length) {
                                $expected_length = $res->content_length || 0;
                             }
                             if ($expected_length) {
                                  printf STDERR "%d%% - ",
                                            100 * $bytes_received / $expected_length;
                             }
                             print STDERR "$bytes_received bytes received\n";
      
                             # XXX Should really do something with the chunk itself
                             # print $chunk;
                         });
             print $res->status_line, "\n";
      
        oops. accidentally hit the 'save' key.. so combining the above with anonymous code.. you can do
        my $req = $mech->click_button(name => 'begin');
        $mech->request( $req, '/PATH/TO/SAVED/FILE' );
        
        request method in W::M is an overload of LWP::UserAgent's request method.