perlmonkdr has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks!!! I have a little problem with LWP if I use this inside a loop for takes pages:

my $res = $ua->get("http://www.perlmonks.com");
if ($res->is_success) {
}

[download]

after some loops it returns an Error 500 timeout, but if I use this:

my $req = HTTP::Request->new(GET => "http://www.perlmonks.com");
my $res = $ua->request($req);
if ($res->is_success) {
}

[download]

Works properly.

I saw other post with same problem and LWP errors reports but not solution is posted.

Thank a lot

Replies are listed 'Best First'.
Re: LWP error 500 timeout
by ikegami (Patriarch) on Feb 16, 2008 at 19:48 UTC
    Assuming $ua is a LWP::UserAgent object, those two snippets are identical. The call to get simply ends up in the exact same call to request, nothing more.
[reply]
[d/l]
[select]

      Thank for your reply, but the first method fail after some request in a loop and the second not, what may happen?

[reply]

        I don't buy that causal relationship. I believe the problem is not related to the snippet you use and that your barking up the wrong tree. I could be wrong, but I don't see any difference between those two snippets.

[reply]
Re: LWP error 500 timeout
by proceng (Scribe) on Feb 16, 2008 at 23:25 UTC
    1) Does this happen on one site or multiple sites?

    2) In one of your replies, you show a tight loop without any chance for the site to recover. Think about adding a sleep interval before resubmitting the request.

    3) Be aware that most sites will throttle you if you send too many requests (search the web for Denial of Service).

    4) You said you "saw other post with same problem". Do not expect us to search for the post, reference it within your message.

    5) The problem could well be with your ISP, rather than perl ;-) .

    6) Please read the replies to your post prior to saying that we did not solve your problem.

[reply]
Re: LWP error 500 timeout
by Khen1950fx (Canon) on Feb 16, 2008 at 20:18 UTC
    For example, a call to get:

    #!/usr/bin/perl

use strict;
use warnings;
use HTTP::Response;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new;
my $res = $ua->get('http://www.perlmonks.com');

if ($res->is_success){
    print $res->decoded_content;
}
else {
    print "Error: " . $res->status_line . "\n";
}

    [download]
[reply]
[d/l]

      Yes, thank for your reply, but note, that this method fail after some request and using the second way not, Why?

      #!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;
use LWP::Debug qw(+);

BEGIN {
 close STDERR;
 open STDERR, '>./lwp-error.txt';
}

END { close STDERR; }

my $a=0;
my $ua = LWP::UserAgent->new;

# in some point this method fail.
while (11000 > $a++) {
my $res = $ua->get('http://www.perlmonks.com');

if ($res->is_success){
    print $res->decoded_content;
}
else {
    print "Error: " . $res->status_line . "\n";
}

}

      [download]
[reply]
[d/l]
        Phases of the moon.
[reply]
[reply]

Back to Seekers of Perl Wisdom