LWP 500 read timeout

bharat88 has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks, My query is similar to : http://www.perlmonks.org/?node_id=781476

I have basically written a code to download a file from the net. The link works when tried from a browser.

But, not using the below script:


#!/usr/local/bin/perl -w
use strict;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);

if($res->is_success){
        print $res->is_success."\n";
        my @data = split '\n', $res->content;
        foreach my $line (@data){
                print $line."\n";
        }
}
else{

        my @data = split '\n', $res->content;
        print $res->status_line . "\n";
        print $res->as_string . "\n";
        my $err =  sprintf "(%d)(%s)(%d)(%s)\n", $!,$!,$^E,$^E;
        print "Err: $err\n";
        print "\n\n\n\\n\n\n";
        for my $i(0 .. 256){
                $! = $i;
                $^E = $i;
                printf "(%d)(%s)(%d)(%s)\n", $!,$!,$^E,$^E;

        }
}
[download]

The only problem is that I cannot mention the URL used.

Here is the output:
500 read timeout

500 (Internal Server Error) read timeout

Content-Type: text/plain

Client-Date: Tue, 15 Jun 2010 12:19:02 GMT

Client-Warning: Internal response

500 read timeout

Err: (0)()(0)()

Any idea if this is a server side problem, or a client-side? Sure looks like a sever side, but the link works externally. Also, the file is being downloaded if I use curl.

Thanks, Bharat.

Comment on LWP 500 read timeout Download Code

Replies are listed 'Best First'.
Re: LWP 500 read timeout by duff (Parson) on Jun 15, 2010 at 13:27 UTC
Um ... I don't see where you set `$url` in your code. And it looks like you have warnings and stricture turned on, so you should get an appropriate error message for not declaring `$url`. duff	[reply] [d/l] [select]
Re^2: LWP 500 read timeout by bharat88 (Initiate) on Jun 15, 2010 at 14:57 UTC
Hey duff, Like I said, I can't mention the URL. But I have defined it in my code.	[reply]
Re: LWP 500 read timeout by ahmad (Hermit) on Jun 16, 2010 at 01:17 UTC
My guess, You forgot to load HTTP::Request module. Which is not required for your script and I have absolutely no idea why you're using it while the only thing you need is already included in LWP::UserAgent module a simple `$ua->get` would the job just fine, you don't need to build a request yourself.	[reply] [d/l]
Re^2: LWP 500 read timeout by Anonymous Monk on Jun 16, 2010 at 05:07 UTC
Hi ahmad, It doesn't work even if I use a simple $ua->get($url). Basically, using perl and LWP::UserAgent or LWP::Simple, it doesn't work. It works with curl ,but I cannot install that on the box that I'm working on. I need to make it work using standard Perl modules. Any other way, I can download files from the net, using Perl? Thanks, Bharat.	[reply]
Re^3: LWP 500 read timeout by bharat88 (Initiate) on Jun 16, 2010 at 05:30 UTC
It works using wget too. But I still need to rectify what is going wrong with the Perl script. Thanks, Bharat.	[reply]
Re^2: LWP 500 read timeout by Anonymous Monk on Jun 16, 2010 at 04:07 UTC
My guess, You forgot to load HTTP::Request module. If that were the case, his program would die. This is not the case. `$ perl -MLWp -le"print for grep /HTTP/, keys %INC" HTTP/Status.pm HTTP/Date.pm HTTP/Response.pm HTTP/Message.pm HTTP/Request.pm HTTP/Headers.pm` [download]	[reply] [d/l]
Re: LWP 500 read timeout by Anonymous Monk on Jun 16, 2010 at 08:24 UTC
"Client-Warning: Internal response" indicates that LWP generated the error, rather than the server, and "500 read timeout" indicates that the HTTP request timed out on the client side. Try increasing the timeout, eg with `$ua->timeout($seconds)`. The default appears to be 180.	[reply] [d/l]
Re^2: LWP 500 read timeout by Anonymous Monk on Jun 16, 2010 at 09:09 UTC
Increased the timeout to 1000s and tried. It still didn't work. Since curl and wget work, maybe it isn't a server side problem. Can it be a problem related to headers not being allowed,etc? Can I remove such default headers? Thanks, Bharat.	[reply]
Re^3: LWP 500 read timeout by Anonymous Monk on Jun 16, 2010 at 09:42 UTC
Can it be a problem related to headers not being allowed,etc? Can I remove such default headers? It could be firewall, proxy, or anything. Check. Get WireShark/Ethereal and compare	[reply]
Re: LWP 500 read timeout by Gangabass (Vicar) on Jun 15, 2010 at 13:32 UTC
May be your browser use proxy?	[reply]
Re^2: LWP 500 read timeout by bharat88 (Initiate) on Jun 15, 2010 at 14:59 UTC
Hey Gangabass, No proxy involved. If I try the following: `curl -o file.out $url`; It works fine.	[reply] [d/l]
Re^3: LWP 500 read timeout by Anonymous Monk on Jun 14, 2016 at 12:52 UTC
Hi, Sorry to up something old... I have the same problem.. how did you fix that? Thanks	[reply]