cormanaz has asked for the wisdom of the Perl Monks concerning the following question:

Good day Bros. So I am trying to port a script from windoze (that works fine there) to Linux. It runs a Google search and gets the results using LWP. When I run it, I immediately get a response with response code 500, response msg "read timeout," and a response header with key "client-warning" with value "Internal Response." This seems to have something to do with the request url I am sending because I get the same thing when I try to access the Perlmonks seekers page, i.e. this
use LWP::UserAgent; my $request_url = 'http://perlmonks.com'; my $user_agent = LWP::UserAgent->new; $user_agent->timeout(30); my $request = HTTP::Request->new('GET', $request_url); my $response = $user_agent->request($request);
generates the same error. Anyone know what's going on? I googled it and super searched it and the only other reports I saw with this error seem to involve LPW::UserAgent::Parallel, which I'm not using.

TIA...

Steve

UPDATE

Eureka! All along I had been running the code in ActiveState Komodo IDE 4.0. I finally took to stepping though the code with its debugger. I narrowed the problem down to line 313 of http.pm, which leads to a socket read on line 232 of Methods.pm, which calls sub sysread in line 382 of http.pm. If I stepped through the code slowly it would return a valid read on perlmonks.com, but if I set breakpoints on line 313 and on sub sysread and hit continue to go between them, I would get the error. Since there seemed to be an interaction between debugger actions and the error, I tried it from the command line and it worked. I guess there was some kind of bad interaction between Komodo and the perl interpreter.

Why this error occurs for perlmonks but not google is a mystery to me, but there you have it. I am going to put in a bug report on Komod and go my merry way.

Thanks for all your suggestions.

Steve

Replies are listed 'Best First'.
Re: LWP error 500 read timeout "internal response"
by kyle (Abbot) on May 17, 2007 at 01:49 UTC

    When I run it (on Linux), I get a successful response. Can you access the web using a tool like wget or curl? Is there a proxy involved (or should there be one)?

      When I run wget http://www.perlmonks.com from the command line, it works. When I try to get the same URL from the script, I get the stated error. When I try http://google.com with the script it works! There is no proxy.

        Have you tried changing the URL to http://localhost (This is assuming you have a web server running on the localhost) ? This would tell you if it is a problem with the code versus a problem with the network.

        If this is still a problem my assumption would that the module installation is Foo-Bared rather than the code. There might an incompatibility with local libraries or the the version of Perl running on that box(which has happened to me before).

Re: LWP error 500 read timeout "internal response"
by kyle (Abbot) on May 17, 2007 at 02:18 UTC

    After a look through the LWP::UserAgent source, it seems that "Client-Warning: Internal response" is something that it produces in various error conditions. I invoked it by mangling the URL in your code. In that case, print $response->as_string, "\n" gave me this:

    501 Protocol scheme 'x' is not supported Content-Type: text/plain Client-Date: Thu, 17 May 2007 02:13:56 GMT Client-Warning: Internal response 501 Protocol scheme 'x' is not supported

    So you might get a little more info from that or just status_line.

      Here's the dumper of the response
      $VAR1 = bless( { '_content' => '500 read timeout ', '_rc' => 500, '_headers' => bless( { 'client-warning' => 'Internal +response', 'client-date' => 'Thu, 17 May +2007 11:06:00 GMT', 'content-type' => 'text/plain' }, 'HTTP::Headers' ), '_msg' => 'read timeout', '_request' => bless( { '_content' => '', '_uri' => bless( do{\(my $o = +'http://www.perlmonks.com')}, 'URI::http' ), '_headers' => bless( { 'user-a +gent' => 'libwww-perl/5.805' }, 'HTTP: +:Headers' ), '_method' => 'GET' }, 'HTTP::Request' ) }, 'HTTP::Response' );

        A "read timeout" is just a regular timeout. You could try changing your "$user_agent->timeout(30)" to "$user_agent->timeout(300)" or something and see if that helps. Either your network is slow (or failed) or the site is not responding fast enough.

        hmmm ... how come the uri from the response object doesn't match the request_url from the OP?

        -derby
Re: LWP error 500 read timeout "internal response"
by varian (Chaplain) on May 17, 2007 at 07:12 UTC
    You can trace LWP activities to get more details on where it gets stuck, to do so just insert another use line:
    use LWP::Debug qw(+);
    I cannot copy the problem it runs fine on my Linux system, it appears to be an environment or system security (SELinux?) related issue.
Re: LWP error 500 read timeout "internal response"
by Khen1950fx (Canon) on May 17, 2007 at 12:05 UTC
    I couldn't get your script to work for me; however, I made some modifications, and this worked---I'm on FC6 with targeted SELinux:

    #!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; my $req; my $res; my $ua = LWP::UserAgent->new; $ua->agent("Firefox/1.5.0.10"); $ua->timeout(30); $req = HTTP::Request->new(GET => 'http://www.perlmonks.org/?node=Seeke +rs%20of%20Perl%20Wisdom'); $res = $ua->request($req); if ($res->is_success) { print $res->content; } else { print "Error: " . $res->status . "\n"; }
Re: LWP error 500 read timeout "internal response"
by Anonymous Monk on May 20, 2007 at 09:53 UTC
    Eureka! All along I had been running the code in ActiveState Komodo IDE 4.0.
    Please remember to disclose important information
      While Komodo is the best debugger I've ever used for Perl, it has been my experience that it will report "500 read timeout" HTTP responses when running Linux LWP or WWW::Mechanize scripts. If you run the same script from the command line, you will not see these problems (it just sucks to be you if you're trying to debug a problem that occurs somewhere after a few HTTP requests). This has been my experience for Komodo version 3.5 and 4.0. Let's hope the ActiveState team can fix this sometime soon, this is the only fault I can find in their otherwise *excellent* tool.
        Blame/credit me for most of the Perl debugger in Komodo. The part that talks to Perl is based on the 5.8 perl5db.pl library, but it differs quite a bit, given that it's a graphical debugger. I've seen this bug come and go over the years. I can't repro this particular bug on Windows, but certainly can on Linux, but only with certain sites (perlmonks.org yes, wired.com no). The debugger doesn't override any core routines. Any insight as to what's going on would be a great help. The timeout occurs on the first attempt to read from the socket. Doing something like this doesn't make a difference: $ua->timeout(1_000_000) if $^P; # has no effect