m3LLow has asked for the wisdom of the Perl Monks concerning the following question:

I have written a form submission script which uses LWP::UA and proxy servers. If operation was successful it logs it, and counts one more successful submission. After some time I have noticed that some of the proxies are really slow and while $ua is grepping the page, $ua->timout ceases the operation. Howerver, despite of form was submitted, $res->is_success has negative value.

Now let's go to the point. I also have noticed what it's enough to get just a first line of server responce, compare it with a line in a script and if they match than we can log it and jump to the next proxy server. How to do this without "if ($res->is_success)"?

Chunk of shortened code from my script:
<code>use LWP::UserAgent;
$ua = new LWP::UserAgent;
$ua->proxy('http',"http://".$proxy);
$ua->timeout(5);
$req = new HTTP::Request POST => 'http://any.com/cgi-bin/any.pl';
$req->content('name=value');
$res = $ua->request($req);
if ($res->is_success) {...

Replies are listed 'Best First'.
Re: form submission with LWP::UA using proxies
by Nightblade (Beadle) on Jul 18, 2002 at 13:14 UTC
    You can to analyze error message
    use LWP::UserAgent; $ua = new LWP::UserAgent; $ua->proxy('http',"http://".$proxy); $ua->timeout(5); $req = new HTTP::Request POST => 'http://any.com/cgi-bin/any.pl'; $req->content('name=value'); $res = $ua->request($req); $err_msg=$res->error_as_HTML(); if($err_msg =~ m/500 read timeout/) { #set longer timeout and retry submition or jump to next proxy } else {...
      better way to do this work:
      use LWP::UserAgent; $ua = new LWP::UserAgent; $ua->proxy('http',"http://".$proxy); $ua->timeout(5); $req = new HTTP::Request POST => 'http://any.com/cgi-bin/any.pl'; $req->content('name=value'); $res = $ua->request($req); if(substr($res->status_line(),0,3) eq "500") { #set longer timeout and retry submition or jump to next proxy } else {...
      Thank you, NB ;)

      Actually timout is not a problem in this script. The main point is howto optimise and shorten the process. Instead of grepping all submission response thru a slow proxy, I just want to grep a very first line of it and move to next proxy server. Howto do this? This is the main problem ;-)
        So use a callback on the response. From lwpcook:
        Or you can process the document as it arrives (second $ua->request() argument is a code reference): use LWP::UserAgent; $ua = LWP::UserAgent->new; $URL = 'ftp://ftp.unit.no/pub/rfc/rfc-index.txt'; my $expected_length; my $bytes_received = 0; my $res = $ua->request(HTTP::Request->new(GET => $URL), sub { my($chunk, $res) = @_; $bytes_received += length($chunk); unless (defined $expected_length) { $expected_length = $res->content_length | +| 0; } if ($expected_length) { printf STDERR "%d%% - ", 100 * $bytes_received / $expe +cted_length; } print STDERR "$bytes_received bytes received +\n"; # XXX Should really do something with the ch +unk itself # print $chunk; }); print $res->status_line, "\n";
        I've used this to get the information from the header without having to spend time fetching all the body. You just have to die when you get enough, as it says in LWP::UserAgent:
        The subroutine variant requires a reference to callback routine as the second argument to the request method and it can also take an optional chuck size as the third argu- ment. This variant can be used to construct "pipe-lined" processing, where processing of received chuncks can begin before the complete data has arrived. The callback func- tion is called with 3 arguments: the data received this time, a reference to the response object and a reference to the protocol object. The response object returned from the request method will have empty content. If the request fails, then the the callback routine is not called, and the response->content might not be empty. The request can be aborted by calling die() in the call- back routine. The die message will be available as the "X-Died" special response header field.

        -- Randal L. Schwartz, Perl hacker

        I dont think, what LWP can solve this problem, you can use IO::Socket module.
        use IO::Socket; $sock = IO::Socket::INET->new(PeerAddr => $proxy_server, PeerPort => $proxy_port, Proto => 'tcp'); send($sock,"GET $url HTTP/1.0\n\n",''); recv($sock,$answer,$length,''); if($answer =~ m/.../) { ...