ttlgreen has asked for the wisdom of the Perl Monks concerning the following question:

Hello again perlmonks!

Here I am with another problem that has me running out of things to throw at my computer.

For some reason I can't seem to do a GET request using LWP::UserAgent

Watching the packets I see it DOES actually send the request, but before the response comes back it starts sending RST packets and then it ends.

I've been through the entire doc on cpan for this class 10 times now and it even has the same behaviour if I copy and paste the example code from the docs as well! It also does it for any URL I try.

The is_success() method always returns true, however as the 200-Success packet never arrives I don't understand why it returns true. Also content(), decoded_content(), and even as_string() all return blank.

This is an exact copy of the code from the docs that gives me this behaviour and everything else I've tried does as well: 

 require LWP::UserAgent;
 
 my $ua = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 
 my $response = $ua->get('http://search.cpan.org/');
 
 if ($response->is_success) {
     print $response->decoded_content;  # or whatever
 }
 else {
     die $response->status_line;
 }

[download]

Could there be something wrong with my perl or libwww installation? I reinstalled both (and I'm on Gentoo which means building them from the source). I had to rebuild perl to add the iThreads functionality and then built libwww again as a last resort to try to solve this problem.

POST requests seem to go through just fine although I still don't get anything in the content() or decoded_content(), I'm not sure if that is related. The 200/Success packet does come back with the expected body data though unlike the GET requests.

I'm completely at a loss for ideas. Any help would be greatly appreciated.

Replies are listed 'Best First'.
Re: LWP::UserAgent not finishing get requests?
by ikegami (Patriarch) on Jan 17, 2009 at 00:59 UTC

    I'd start by eliminating variables. Specifically, comment out $ua->timeout(10); and $ua->env_proxy;.

    even as_string() all return blank.

    That's impossible! HTTP::Response objects are guaranteed to return "000 Unknown code" at the very least. 

    sub status_line
{
    my $self = shift;
    my $code = $self->{'_rc'}  || "000";
    my $mess = $self->{'_msg'} || HTTP::Status::status_message($code) 
+|| "Unknown code";
    return "$code $mess";
}

sub as_string
{
    require HTTP::Status;
    my $self = shift;
    my($eol) = @_;
    $eol = "\n" unless defined $eol;

    my $status_line = $self->status_line;
    my $proto = $self->protocol;
    $status_line = "$proto $status_line" if $proto;

    return join($eol, $status_line, $self->SUPER::as_string(@_));
}

    [download]

    And besides, we know that _rc is ≥200 and <300 because is_success returned true.

[reply]
[d/l]
[select]
      This is a total guess... but maybe it's empty because the response isn't actually getting back from the server?

      Those darn RST packets coming from my script for no apparent reason seem to do their job well...

[reply]
        Look at the code I pasted. No matter what the object contains, $response->as_string will always return something. And by your own testimony, is_success returns true so it's not empty.
[reply]
[d/l]
[select]
Re: LWP::UserAgent not finishing get requests?
by derby (Abbot) on Jan 17, 2009 at 00:32 UTC

    Sounds like a misconfigured proxy to me. This code: 

    #!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;

my $response = $ua->get('http://search.cpan.org/');

if ($response->is_success) {
  print $response->content;
} else {
  die $response->status_line;
}

    [download]
    gives me the search.cpan.org page. What happens when you use a browser to retrieve the page? What about curl or wget?

    -derby
[reply]
[d/l]
      Hmm I don't have a proxy. Does that mean I shouldn't have the $ua->env_proxy; there? I just tried the code without that which didn't help, and also without the timeout setting.

      Still the same result... Nothing..

      I'm so confused

      EDIT: Sorry I forgot to say I have no problems whatsoever from normal browsers or wget... It's definetly isolated to my script. For that matter the LWP::Simple::get works just fine. I'd just use it but I need the cookie and POSTing capabilities of the full UserAgent class.

[reply]
[d/l]

        Your script? What about the simple snippet you posted? That too? What's this give you? 

        !#/usr/bin/perl

use strict;
use warnings;

use Data::Dumper;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

my $response = $ua->get('http://search.cpan.org/');

print Dumper( $response );

        [download]

        -derby
[reply]
[d/l]
        Hmm I don't have a proxy.

        Are you sure? Lots of networks (my ISP included) do transparent proxying to make more efficient use of bandwidth (among other reasons).

        Life is denied by lack of attention,
        whether it be to cleaning windows
        or trying to write a masterpiece...
        -- Nadia Boulanger
[reply]

Back to Seekers of Perl Wisdom