in reply to LWP::SIMPLE fails on certain URL

In situations like these, you must use LWP::Debug 
$>perl -MLWP::Debug=+ -MLWP::Simple -we"print
 get('http://wire.ap.org/APnews/center_minor.html?FRONTID=SCIENCE')"
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::request: Simple response: Bad Request
Use of uninitialized value in print at -e line 1.

[download]
a simple LWP::Simple::get($url) didn't work for me either, even though I could see the page in me browser, but a LWP::Simple::getstore did, so I debugged that too 
$>perl -MLWP::Debug=+ -MLWP::Simple -we"print
 getstore('http://wire.ap.org/APnews/center_minor.html?FRONTID=SCIENCE
+','file.tx
t')"
LWP::UserAgent::new: ()
LWP::UserAgent::request: ()
LWP::UserAgent::simple_request: GET http://wire.ap.org/APnews/center_m
+inor.html?
FRONTID=SCIENCE
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::http::request: GET /APnews/center_minor.html?FRONTID=SC
+IENCE HTTP
/1.0
Host: wire.ap.org
User-Agent: LWP::Simple/5.51

LWP::Protocol::http::request: reading response
LWP::Protocol::http::request: HTTP/1.1 302 Found
Date: Sun, 27 Jan 2002 10:36:34 GMT
Server: Apache/1.3.12 (Unix) mod_perl/1.23
Location: /public_pages/WirePortal.pcgi
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>302 Found</TITLE>
</HEAD><BODY>
<H1>Found</H1>
The document has moved <A HREF="/public_pages/WirePortal.pcgi">here</A
+>.<P>
<HR>
<ADDRESS>Apache/1.3.12 Server at wire.ap.org Port 80</ADDRESS>
</BODY></HTML>
LWP::Protocol::http::request: HTTP/1.1 302 Found
LWP::Protocol::collect: read 277 bytes
LWP::UserAgent::request: Simple response: Found
LWP::UserAgent::request: ()
LWP::UserAgent::simple_request: GET http://wire.ap.org/public_pages/Wi
+rePortal.p
cgi
LWP::UserAgent::_need_proxy: Not proxied
LWP::Protocol::http::request: ()
LWP::Protocol::http::request: GET /public_pages/WirePortal.pcgi HTTP/1
+.0
Host: wire.ap.org
User-Agent: LWP::Simple/5.51

LWP::Protocol::http::request: reading response
LWP::Protocol::http::request: HTTP/1.1 200 OK
Connection: close
Date: Sun, 27 Jan 2002 08:31:02 GMT
Server: Apache/1.3.12 (Unix) mod_perl/1.23
Content-MD5: MA6SFpj03gSVT0/n/+ZEnA
Content-Type: text/html; charset=ISO-8859-1
Title: Testing JavaScript

<!-- pcache/1.7.3 Cache loaded from file on warrant Sun Jan 27 03:31:0
+6 2002 -->


<!-- $Id: WirePortal.pcgi,v 1.10 2001/12/13 21:04:53 jxu Exp $  -->
<HTML>
<HEAD>
<META HTTP-EQUIV=Refresh CONTENT="1; URL=/public_pages/WirePortal.pcgi
+/nojs.html
">
<TITLE>Testing JavaScript</TITLE>
<script language="JavaScript">
<!--
function gogo() {
self.location.href='/public_pages/WirePortal.pcgi/us_portal.html'
}

// -->
</script>
<BODY BGCOLOR="#FFFFFF" onload="gogo()">
</BODY>
</HTML>
LWP::Protocol::http::request: HTTP/1.1 200 OK
LWP::Protocol::collect: read 478 bytes
LWP::UserAgent::request: Simple response: OK
200

[download]
I'd definetly say this is a bug in LWP::UserAgent, a ++ to anyone who takes the time and figures out where it is.

Below is sub LWP::UserAgent::request, which is where LWP::Simple::get seems to fail. But first, here is the line in the debug line in sub request which gives us the interesting error 

    LWP::Debug::debug('Simple response: ' .
              (HTTP::Status::status_message($code) ||
               "Unknown code $code"));

[download]
=item $ua->request($request, $arg [, $size])

Process a request, including redirects and security.  This method may
actually send several different simple requests.

The arguments are the same as for C<simple_request()>.

=cut

sub request
{
    my($self, $request, $arg, $size, $previous) = @_;

    LWP::Debug::trace('()');

    my $response = $self->simple_request($request, $arg, $size);

    my $code = $response->code;
    $response->previous($previous) if defined $previous;

    LWP::Debug::debug('Simple response: ' .
              (HTTP::Status::status_message($code) ||
               "Unknown code $code"));

    if ($code == &HTTP::Status::RC_MOVED_PERMANENTLY or
    $code == &HTTP::Status::RC_MOVED_TEMPORARILY) {

    # Make a copy of the request and initialize it with the new URI
    my $referral = $request->clone;

    # And then we update the URL based on the Location:-header.
    my $referral_uri = $response->header('Location');
    {
        # Some servers erroneously return a relative URL for redirects
+,
        # so make it absolute if it not already is.
        local $URI::ABS_ALLOW_RELATIVE_SCHEME = 1;
        my $base = $response->base;
        $referral_uri = $HTTP::URI_CLASS->new($referral_uri, $base)
                    ->abs($base);
    }

    $referral->url($referral_uri);

    return $response unless $self->redirect_ok($referral);

    # Check for loop in the redirects
    my $count = 0;
    my $r = $response;
    while ($r) {
        if (++$count > 13 ||
                $r->request->url->as_string eq $referral_uri->as_strin
+g) {
        $response->header("Client-Warning" =>
                  "Redirect loop detected");
        return $response;
        }
        $r = $r->previous;
    }

    return $self->request($referral, $arg, $size, $response);

    } elsif ($code == &HTTP::Status::RC_UNAUTHORIZED ||
         $code == &HTTP::Status::RC_PROXY_AUTHENTICATION_REQUIRED
        )
    {
    my $proxy = ($code == &HTTP::Status::RC_PROXY_AUTHENTICATION_REQUI
+RED);
    my $ch_header = $proxy ?  "Proxy-Authenticate" : "WWW-Authenticate
+";
    my @challenge = $response->header($ch_header);
    unless (@challenge) {
        $response->header("Client-Warning" => 
                  "Missing Authenticate header");
        return $response;
    }

    require HTTP::Headers::Util;
    CHALLENGE: for my $challenge (@challenge) {
        $challenge =~ tr/,/;/;  # "," is used to separate auth-params!
+!
        ($challenge) = HTTP::Headers::Util::split_header_words($challe
+nge);
        my $scheme = lc(shift(@$challenge));
        shift(@$challenge); # no value
        $challenge = { @$challenge };  # make rest into a hash
        for (keys %$challenge) {       # make sure all keys are lower 
+case
        $challenge->{lc $_} = delete $challenge->{$_};
        }

        unless ($scheme =~ /^([a-z]+(?:-[a-z]+)*)$/) {
        $response->header("Client-Warning" => 
                  "Bad authentication scheme '$scheme'");
        return $response;
        }
        $scheme = $1;  # untainted now
        my $class = "LWP::Authen::\u$scheme";
        $class =~ s/-/_/g;
    
        no strict 'refs';
        unless (%{"$class\::"}) {
        # try to load it
        eval "require $class";
        if ($@) {
            if ($@ =~ /^Can\'t locate/) {
            $response->header("Client-Warning" =>
                      "Unsupported authentication scheme '$scheme'");
            } else {
            $response->header("Client-Warning" => $@);
            }
            next CHALLENGE;
        }
        }
        return $class->authenticate($self, $proxy, $challenge, $respon
+se,
                    $request, $arg, $size);
    }
    return $response;
    }
    return $response;
}

[download]
update: I have $LWP::Simple::VERSION = 1.33; and $LWP::VERSION = 5.51;

 
______crazyinsomniac_____________________________
Of all the things I've lost, I miss my mind the most.
perl -e "$q=$_;map({chr unpack qq;H*;,$_}split(q;;,q*H*));print;$q/$q;"

Replies are listed 'Best First'.
Re: (crazyinsomniac) Re: LWP::SIMPLE fails on certain URL
by shotgunefx (Parson) on Jan 28, 2002 at 03:15 UTC
    It is a known bug. (see post below.) I think it is a socket library problem. Removing the timeout in _trivial_get fixes it. Also, prepending a space will force it to not use the _trivial_get which will also get around it.

    I'll ++everything the person who fixes this writes. I spent over a month trying to figure it out. Drove me #!@#! nuts!

    -Lee

    "To be civilized is to deny one's nature."
[reply]

In Section Seekers of Perl Wisdom