LWP - 500 Internal Error

oalvi has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have the following code which I'm using for monitoring URLs:

# !C:\Perl\bin

use strict;
use warnings;
use LWP::UserAgent;

#my $wsr = LWP::UserAgent -> new;
#$wsr -> timeout( 20 );

my $url       = shift or die "URL expected\n";
my $useragent = LWP::UserAgent->new;

$useragent->proxy(['http', 'ftp','https'], 'http://192.168.5.5:8080/')
+;
$useragent -> timeout( 20 );

my $request   = HTTP::Request->new( HEAD => $url );
my $response  = $useragent->request($request);

print $response->status_line, "\n";
[download]

It works fine for most URLs, http or https. However for some URLs, which function fine from the explorer, I get errors as follows:

https://elmag.elm.com.sa

500 Internal Server Error

https://www.etv.com.sa

500 Internal Server Error

Other https addresses work fine, for example

https://eserve.com.sa

200 OK

Looking forward to your support and assistance.

Comment on LWP - 500 Internal Error Download Code

Replies are listed 'Best First'.
Re: LWP - 500 Internal Error by ig (Vicar) on Sep 01, 2009 at 14:18 UTC
Can you review the server error logs to find the causes of the errors? You can use a network sniffer (e.g. wireshark) to capture the traffic between your system and the server. You might find some clues by comparing successful and failed accesses. Even with HTTPS, not everything is encrypted. There are tools that might allow you to inspect the HTTPS exchange with the server. If Firefox browser works, you might be able to use some of its debugging features to find out what is happening. Otherwise, there are proxy servers which might allow you to inspect the exchange (e.g. fiddler. Ultimately, you could brute force decrypt the traffic, thought this might take some time. Might cookies be required by the server?	[reply]
Re: LWP - 500 Internal Error by Anonymous Monk on Sep 01, 2009 at 12:29 UTC
See Re^8: LWP for URL monitoring - 500 connect timeout errors	[reply]
Re^2: LWP - 500 Internal Error by oalvi (Novice) on Sep 01, 2009 at 13:25 UTC
Hi and Thank for the link, Was my thread, vry interesting too - but i didn't think I could start from there anew. Anyway, I tried adding the sfprint debug, doesn't work for me yet. # !C:\Perl\bin use strict; use warnings; use LWP::UserAgent; @ARGV = qw[ http://www.eserve.com.sa/ ]; my $url = shift or die "URL expected\n"; my $useragent = LWP::UserAgent->new; $useragent->proxy(['http', 'ftp','https'], 'http://192.168.5.5:8080/') +; $useragent -> timeout( 20 ); my $request = HTTP::Request->new( GET => $url ); my $response = $useragent->request($request); my $err = sprintf "(%d)(%s)(%d)(%s)\n", $!,$!,$^E,$^E; print $useragent->proxy('http'),"\n"; print $response->status_line, "\n"; print "Err : $err \n"; [download] I get this error `http://192.168.5.5:8080/ 200 OK Err : (9)(Bad file descriptor)(0)()` [download] Seems it ran everything alrights, something simple here I must be missing	[reply] [d/l] [select]
Re^3: LWP - 500 Internal Error by Anonymous Monk on Sep 01, 2009 at 13:39 UTC
200 OK means there was no error	[reply]
Re^4: LWP - 500 Internal Error by Anonymous Monk on Sep 01, 2009 at 19:35 UTC
Re^5: LWP - 500 Internal Error by Anonymous Monk on Sep 01, 2009 at 21:41 UTC
Some notes below your chosen depth have not been shown here
Re: LWP - 500 Internal Error by Anonymous Monk on Jul 07, 2010 at 08:33 UTC
You should set 'agent' parameter for your userAgent object. It seems that some sites verify user_agent field in Request and reject connection if it is empty or differs from common variants like 'mozilla', 'opera', etc >> $useragent -> agent( 'Mozilla' );	[reply]