Re: Timing out POE http client

Hey. First, you should also try contacting the author. I mainly read the site via RSS, and the sheer number of posts guarantees I miss all but the last 12 at any given time. I would have missed your post if someone hadn't pointed it out to me.

The problem is twofold. First, there was a bug in POE::Component::Client::HTTP. Old request timeouts were not removed when the component followed redirects. I was able to find the problem thanks to your test program, so ++ to you. Version 0.74 should be on your favorite CPAN mirror any day now.

The second problem is how POE::Component::Client::HTTP uses POE::Component::Client::Keepalive: simplistically. The short explanation is that unused connections are kept alive longer than you probably want. It's easy enough to work around. Replace the start of your program with this:

use strict;
use warnings;
use Time::HiRes qw( time );
use HTTP::Request;

use POE qw(Component::Client::HTTP Component::Client::Keepalive);

my $start = time;
my $urls_left = 12;

my $cm = POE::Component::Client::Keepalive->new( keep_alive => 1 );

POE::Component::Client::HTTP->spawn(
    Alias     => 'ua',
    Timeout   => shift || 10,
    FollowRedirects => 2,
    Streaming => 0,
    ConnectionManager => $cm,
);
[download]

The new code creates a custom POE::Component::Client::Keepalive with a super-low keep-alive timeout. Unused sockets are discarded after 1 second, so they stop holding program hostage for so long. When POE::Component::Client::HTTP is spawned, it's told to manage connections with the custom Keepalive component rather than simplistically create its own.

These changes work for me, at least with the newly released timeout fix.

1) poerbook:~/projects/poco-client-http% make && perl perlmonks.perl 2
+0
Response (200) Downloads done in: 7.075767993927 seconds.
Run done in: 8.70663499832153 seconds.
[download]

I also removed that warning from the BUGS. Its very presence was a bug. And speaking of bugs, ConnectionManager isn't documented. I'll create a ticket for that at rt.cpan.org and get it in the next release.

-- Rocco Caputo - http://poe.perl.org/

Comment on Re: Timing out POE http client Select or Download Code

Replies are listed 'Best First'.
Re^2: Timing out POE http client by ryantate (Friar) on Mar 30, 2006 at 23:47 UTC
Hello Rocco and thank you for your reply! Next time I will try emailing you directly. I didn't know if this was a common issue. I have a follow up question. Instead of waiting for the (admittedly short) one-second keepalive timeout, could I not call the shutdown method on the ConnectionManager when my results are all back? I know it is in $heap->{cm} in the parent Component::Client::HTTP session, but I do not know how to access this heap. "cm" does not seem to be in the heap available to my response handler.	[reply]
Re^3: Timing out POE http client by rcaputo (Chaplain) on Mar 31, 2006 at 00:38 UTC
Good question. The short answer's no, POE::Component::Client::HTTP doesn't expose the connection manager it uses. It also doesn't have a way to be shut down ahead of time, which hasn't been needed until now. If you're creating your own POE::Component::Client::Keepalive and passing that as the ConnectionManager, however, you still have `$cm` floating around, and you can shut that down without ill effects. Caveat: I've only tried this in your example program, and only when there are no outstanding requests. `-- Rocco Caputo - http://poe.perl.org/`	[reply]