in reply to Using LWP::Parallel

You might want to check out POE::Component::Client::UserAgent. One of the test scripts (t/02-multi.t) does pretty much what you want.

Update 11/03/02: Here is a much shorter script, that at least demonstrates fetching a list of documents, and uses 10 threads. Liberal adding of "print" statements to assist understanding of the code is left as an exercise for the reader.

#!/usr/bin/perl -w use strict; use POE; use POE::Component::Client::UserAgent; my @urls = map { "http://sam.vilain.net/$_" } qw(index.html hobbies.html CV foo bar); sub _start { $_[HEAP]->{alias} = "useragent".$_[ARG0]; POE::Component::Client::UserAgent->new (alias => $_[HEAP]->{alias}); $_[KERNEL]->yield("next"); } sub next { if (my $url = pop @urls) { $_[KERNEL]->post ( $_[HEAP]->{alias} => "request", { request => HTTP::Request->new(GET => $url), response => $_[SESSION]->postback('next') } ); } else { $_[KERNEL]->post ( $_[HEAP]->{alias} => "shutdown", ); } } for (1..10) { POE::Session -> create ( inline_states => { _start => \&_start, next => \&next, }, args => [ $_ ], # arguments ); } $poe_kernel->run();
PS LWP::Parallel::UserAgent 2.51 has a bug;
  1. The line 182 of LWP::Parallel::UserAgent.pm needs to be my $self = new LWP::UserAgent; instead of my $self = new LWP::UserAgent $init;
  2. You may also need to insert the line use HTML::HeadParser; anywhere near the top of LWP::Parallel::Protocol.pm.