in reply to Re: form submission with LWP::UA using proxies
in thread form submission with LWP::UA using proxies

Thank you, NB ;)

Actually timout is not a problem in this script. The main point is howto optimise and shorten the process. Instead of grepping all submission response thru a slow proxy, I just want to grep a very first line of it and move to next proxy server. Howto do this? This is the main problem ;-)
  • Comment on Re: Re: form submission with LWP::UA using proxies

Replies are listed 'Best First'.
•Re: Re: Re: form submission with LWP::UA using proxies
by merlyn (Sage) on Jul 18, 2002 at 15:24 UTC
    So use a callback on the response. From lwpcook:
    Or you can process the document as it arrives (second $ua->request() argument is a code reference): use LWP::UserAgent; $ua = LWP::UserAgent->new; $URL = 'ftp://ftp.unit.no/pub/rfc/rfc-index.txt'; my $expected_length; my $bytes_received = 0; my $res = $ua->request(HTTP::Request->new(GET => $URL), sub { my($chunk, $res) = @_; $bytes_received += length($chunk); unless (defined $expected_length) { $expected_length = $res->content_length | +| 0; } if ($expected_length) { printf STDERR "%d%% - ", 100 * $bytes_received / $expe +cted_length; } print STDERR "$bytes_received bytes received +\n"; # XXX Should really do something with the ch +unk itself # print $chunk; }); print $res->status_line, "\n";
    I've used this to get the information from the header without having to spend time fetching all the body. You just have to die when you get enough, as it says in LWP::UserAgent:
    The subroutine variant requires a reference to callback routine as the second argument to the request method and it can also take an optional chuck size as the third argu- ment. This variant can be used to construct "pipe-lined" processing, where processing of received chuncks can begin before the complete data has arrived. The callback func- tion is called with 3 arguments: the data received this time, a reference to the response object and a reference to the protocol object. The response object returned from the request method will have empty content. If the request fails, then the the callback routine is not called, and the response->content might not be empty. The request can be aborted by calling die() in the call- back routine. The die message will be available as the "X-Died" special response header field.

    -- Randal L. Schwartz, Perl hacker

Re: Re: Re: form submission with LWP::UA using proxies
by Nightblade (Beadle) on Jul 18, 2002 at 14:15 UTC
    I dont think, what LWP can solve this problem, you can use IO::Socket module.
    use IO::Socket; $sock = IO::Socket::INET->new(PeerAddr => $proxy_server, PeerPort => $proxy_port, Proto => 'tcp'); send($sock,"GET $url HTTP/1.0\n\n",''); recv($sock,$answer,$length,''); if($answer =~ m/.../) { ...
      I think that is what i need! Thank you!