jonp has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone
I am having a problem with reading web pages with my script:
use LWP::UserAgent; 
my $ua = LWP::UserAgent->new(agent=>'Mozilla/4.0 (compatible; MSIE 5.2
+3; Mac_PowerPC)');
my $response = $ua->get('http://www.google.com');
my $page = $response->content;
print HTML $page;

[download]

The weird thing is that this code works about every 10th time that I run it. When it doesn't work I get this error:
"500 Can't connect to www.google.com:80 (connect: Invalid argument)"
I have no idea why it sometimes works and other times it gives this error. If I use the browser, I can always manually open any web page I want. If anyone can help, I would really appreciate it.
Thanks
Jon

Replies are listed 'Best First'.
Re: LWP error 500
by sh1tn (Priest) on Aug 31, 2005 at 23:23 UTC
    I suspect that you are behind firewall or proxy, this is network, but not LWP::UserAgent problem.
    You may want to try the proxy object method: 
    ...
my $ua = LWP::UserAgent->new();

# your proxy ip address instead if '10.0.0.1':
$ua->proxy('http', 'http://10.0.0.1:8080');
...

    [download]


[reply]
[d/l]

      Hi,

      If your proxy asks you for a userid & password you should do the following:

      #!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;

my $ua = new LWP::UserAgent;

$ua->agent("mYpErLsCrIpT.pl/3.0");
$ua->env_proxy;
$ua->proxy(http=>'http://proxy.pxpost:8080');

my $req = new HTTP::Request "GET" => "http://www.slashdot.org";

$req->proxy_authorization_basic('gargle','ImNoTgOnNaTeLl');

print $ua->request($req)->as_string;

      [download]
      --
      if ( 1 ) { $postman->ring() for (1..2); }
[reply]
[d/l]
        I prefer setting up the proxy information in the system environment or by modifying %ENV during the script. This provides maximum compatibility for modules like Crypt::SSLeay which would be required for https access but don't pick up the LWP proxy info.
[reply]

Back to Seekers of Perl Wisdom