I'm trying to use the POE::Component::Client::HTTP module to create some Web client code that can download multiple pages simultaneously.

I'm aware of LWP::Parallel::UserAgent, but I'd like to get to grips with POE, and now seemed like as good a time as any. There isn't much example POE code available yet, although POE's Web site contains some helpful material.

The problem I'm encountering is that I can allocate a set of URLs for a given POE session, but I can't create one pool of URLs which sessions work through together until all URLs have been queried.

Here's some code that spawns 3 sessions, each of which retrieve a page: 

#!/usr/bin/perl -Tw

use strict;

use HTTP::Request::Common qw(GET);
use POE qw(Component::Client::HTTP);

use vars qw(@url);

# Create a POE HTTP client component
POE::Component::Client::HTTP->spawn(
    'Timeout' => 20
);

# Create sessions to be used for HTTP requests
for (1..3) {
    POE::Session->create(
 inline_states => {
     _start  => \&client_start,
     got_response => \&client_got_response
 }
    );
}
$poe__kernel->run();


sub client_start {
    my ($kernel, $heap) = @_[KERNEL, HEAP];
    $kernel->post( 'weeble', 'request',
 got_response => GET 'http://URL_HERE/'
    );
    warn 'STARTED';
}

sub client_got_response {
    my($heap, $request_packet, $response_packet) = @_[HEAP, ARG0, ARG1
+];
    my $http_request  = $request_packet->[0];
    my $http_response = $response_packet->[0];

    print "REQUEST\n",$http_request->headers_as_string();
    print "RESPONSE\n",$http_response->headers_as_string();
}

[download]

To use a pool of URLs, I've removed the $kernel->post call in my client_start subroutine, and placed the following before $poe_kernel->run(): 

foreach my $url (@url) {
    $poe_kernel->post( 'weeble', 'request',
 got_response => GET $url
    );
}

[download]

This fails with the error Can't locate object method "postback" via package "POE::Kernel" at /usr/local/lib/perl5/site_perl/5.005/POE/Component/Client/HTTP.pm line 208. Can anyone help me solve this, maybe using another technique?

In reply to POE HTTP Client problem by tomhukins

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.