Dear Monks,

In a module I have the lines

use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $response = $ua->post('server url', 'some nvp string');
[download]

My problem is that, maybe, 5% of the time the module does not register a response, but the server registers the request as if nothing wrong had ever happened.

Is there anything I might have done to get the module to wait longer for the response?

Anyhow, it was very bad that the request should have been successful, but that the module should, essentially, die at the request line. So I tried this:

use LWPx::ParanoidAgent;
my $ua = LWPx::ParanoidAgent->new;
$ua->timeout(30);
my $response = $ua->post('server url', 'some nvp string');
[download]

In this case, failures occurred more often, say 10% of the time, but at least the request did not register at the server without my receiving a response. Instead, the module received a 500 response "DNS lookup timeout", and the server never received the request.

Still this wasn't a good situ; the module shouldn't give up so easily. So I tried this:

use LWP::UserAgent::Determined;
my $ua = LWP::UserAgent::Determined->new;
$ua->timing( "1,3,15" )
my $response = $ua->post('server url', 'some nvp string');
[download]

My mistake here was to have forgotten my original problem; namely. that I wasn't getting any kind of response with the LWP::UserAgent before execution of the module 'died' so of course the re-try would never occur.

So what I need to do is re-write the LWP::UserAgent::Determined loop by substituting calls to LWPx::ParanoidAgent to force a 500 response if DNS lookup is the hold-up, presumably with shorter timeouts to begin with.

Would it really be that easy? Or is there some consequence I'm overlooking?

Thanks.

more thoughts Having given this a little more thought, I might better frame the problem this way. The server operator wants a 30 second timeout, but the failures I've caught with ParanoidAgent have been DNS lookup timeouts, and 30 seconds seems too long to wait for that, especially since the user is committing money and needs a timely response.

LWPx::ParanoidAgent looks up the host IP this way:

sub _resolve {
    my ($self, $host, $request, ...) = @_;
    my $res = $self->resolver; # from Net::DNS::Resolver
...
    my $sock = $res->bgsend($host) # bgsend from Net::DNS::Resolver
        or die "No sock from bgsend";
    my $rin = '';
    vec($rin, fileno($sock), 1) = 1;
    my $nf = select($rin, undef, undef, $self->_time_remain($request))
+; # _time_remain is the clock for the request
    die "DNS lookup timeout" unless $nf;

    my $packet = $res->bgread($sock) # bgread from Net::DNS::Resolver
        or die "DNS bgread failure";

...
[download]

So it seems like the DNS request could be looped through several tries in a much shorter period, perhaps changing to the secondary name server along the way, while still allowing the subsequent request its 30 seconds.