Hey. First, you should also try contacting the author. I mainly read the site via RSS, and the sheer number of posts guarantees I miss all but the last 12 at any given time. I would have missed your post if someone hadn't pointed it out to me.

The problem is twofold. First, there was a bug in POE::Component::Client::HTTP. Old request timeouts were not removed when the component followed redirects. I was able to find the problem thanks to your test program, so ++ to you. Version 0.74 should be on your favorite CPAN mirror any day now.

The second problem is how POE::Component::Client::HTTP uses POE::Component::Client::Keepalive: simplistically. The short explanation is that unused connections are kept alive longer than you probably want. It's easy enough to work around. Replace the start of your program with this: 

use strict;
use warnings;
use Time::HiRes qw( time );
use HTTP::Request;

use POE qw(Component::Client::HTTP Component::Client::Keepalive);

my $start = time;
my $urls_left = 12;

my $cm = POE::Component::Client::Keepalive->new( keep_alive => 1 );

POE::Component::Client::HTTP->spawn(
    Alias     => 'ua',
    Timeout   => shift || 10,
    FollowRedirects => 2,
    Streaming => 0,
    ConnectionManager => $cm,
);

[download]

The new code creates a custom POE::Component::Client::Keepalive with a super-low keep-alive timeout. Unused sockets are discarded after 1 second, so they stop holding program hostage for so long. When POE::Component::Client::HTTP is spawned, it's told to manage connections with the custom Keepalive component rather than simplistically create its own.

These changes work for me, at least with the newly released timeout fix. 

1) poerbook:~/projects/poco-client-http% make && perl perlmonks.perl 2
+0
Response (200) Downloads done in: 7.075767993927 seconds.
Run done in: 8.70663499832153 seconds.

[download]

I also removed that warning from the BUGS. Its very presence was a bug. And speaking of bugs, ConnectionManager isn't documented. I'll create a ticket for that at rt.cpan.org and get it in the next release.

-- Rocco Caputo - http://poe.perl.org/

In reply to Re: Timing out POE http client by rcaputo
in thread Timing out POE http client by ryantate

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.