Problem with AnyEvent::HTTP

bobr has asked for the wisdom of the Perl Monks concerning the following question:

Hello monks,

I am trying to create simple parallel http getter with AnyEvent::HTTP. My program is bit larger, but following test case demostrates my issue well:

use AnyEvent::HTTP;

my $cv = AnyEvent->condvar;
my @urls = map { chomp; $_ } <DATA>;
for my $url (@urls) {
    $cv->begin;
    http_get $url, sub {
        my ($body, $hdr) = @_;
        warn "[$hdr->{Status}:$hdr->{Reason}] ",$url," (",length($body
+),")\n";
        $cv->end;
    };
}
$cv->recv;

__DATA__
some urls
[download]

This works quite well, except that about 60% of queries end with result 599 - Unknown error. The list of failed ones does not seem to be consistent, it changes with every run. Any clue?

Running ActivePerl 5.10 on WinXP

-- thanks, Roman

Comment on Problem with AnyEvent::HTTP Download Code

Replies are listed 'Best First'.
Re: Problem with AnyEvent::HTTP by OverlordQ (Hermit) on Aug 20, 2010 at 16:13 UTC
Couldn't find 599 - Unknown error anywhere, but there was `Status => 599, Reason => $err`. This was handler for a `AnyEvent::Socket::tcp_connect` call. Since it's Windows, are you running into KB969710? Try checking your event log for Event 4226.	[reply] [d/l] [select]
Re^2: Problem with AnyEvent::HTTP by bobr (Monk) on Aug 20, 2010 at 18:24 UTC
Bingo, this seems to be it. Although I haven't found any 4226 in my event log, it looks like WinXP SP2 is limiting connections to 10. I tried the patch, but it does not seem to work. I found few more places in AnyEvent::HTTP (mostly `on_error` handlers) where reason is taken from outside. To be sure, I added some prefix to those error messages and indeed the error comes from `tcp_connect`. Knowing the limitation, I was able to work around. Not that I like hardcoding the number 10 like this, but limit connections is probably good idea anyway. `use AnyEvent::HTTP; my @urls = map { chomp; $_ } <DATA>; while(my @ten_urls = splice(@urls,0,10)) { my $cv = AnyEvent->condvar; for my $url (@ten_urls) { $cv->begin; http_get $url, timeout => 5, sub { my ($body, $hdr) = @_; warn "[$hdr->{Status}:$hdr->{Reason}] ",$url," (",length($ +body),")\n"; $cv->end; }; } $cv->recv; warn "-----","\n"; }` [download] Really appreciate your help. -- thanks, Roman	[reply] [d/l] [select]
Re: Problem with AnyEvent::HTTP by zentara (Cardinal) on Aug 20, 2010 at 15:00 UTC
Just a thought from a linux user...... maybe AnyEvent is using a pipe somewhere, and on windows, they are deap pipes, and throwing weird errors. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku	[reply]
Re^2: Problem with AnyEvent::HTTP by james2vegas (Chaplain) on Aug 21, 2010 at 10:31 UTC
When AnyEvent needs a pipe it uses portable_pipe which usually maps to pipe, but on Windows it creates a pair of TCP sockets (same with socketpair / portable_socketpair), slow but works better (for AnyEvent's purposes) than pipe on windows.	[reply]
Re^2: Problem with AnyEvent::HTTP by bobr (Monk) on Aug 20, 2010 at 15:22 UTC
Thanks for pointer. I briefly skimmed through the code but I am not all that familiar with HTTP details to understand. The pipes are not only thing broken on Windows ... ;-) -- Roman	[reply]