Simple Coro + AnyEvent::HTTP example?

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I would like to do some HTTP requests in parallel and return the results as soon as all requests are answered or when a timeout occurs. Looks simple and looking at CPAN Coro + AnyEvent::HTTP (or Coro::LWP) should be the right tools for the job.

Only problem is, I am not at all experienced with event loops and the man pages, including inros, had too many "..." to get me going. So my hope is that one of you friendly Perl Monks has a simple example at hand.
I am hoping for something along these lines:

my @all_results;

foreach my $url (@all_urls) {
    http_get $url, sub {
        push @all_results, $_[1];
    };
}
# but how do I know that all results are there or I run into a timeout
+?
# Where and how comes Coro into the picture?
[download]

Many thanks for every help. I just don't get the overall picture and I am sure a tiny example will get me going.

Michael

Comment on Simple Coro + AnyEvent::HTTP example? Download Code

Replies are listed 'Best First'.
Re: Simple Coro + AnyEvent::HTTP example? by Anonymous Monk on Mar 19, 2009 at 08:05 UTC
http://cpansearch.perl.org/src/MLEHMANN/Coro-5.131/eg/lwp	[reply]
Re^2: Simple Coro + AnyEvent::HTTP example? by Anonymous Monk on Mar 19, 2009 at 08:58 UTC
Thanks, that's a good start, I tried to change the example to loop over a few URLs and and seems to work (though it looks odd that Google answers last), what I am still missing is how to set a timeout. Well, I can set it for every LWP connection, but how can I set it for the whole loop? And I don't understand why do I need "$SIG{PIPE} = 'IGNORE';"? So, really many thanks for this valuable pointer but I would like to understand it a bit better and I am still interested in an example with AnyEvent::HTTP. Michael	[reply]
Re: Simple Coro + AnyEvent::HTTP example? by Sinistral (Monsignor) on Mar 19, 2009 at 12:48 UTC
It seems as if you've picked the tools for the job, so I hesitate to make this recommendation, but here goes. Have you checked out the possibility of using LWP::Parallel? It's API is pretty simple, and the examples in the documentation are pretty straightforward. I've used it with a scanner script that checks hundreds of URLs with no problems.	[reply]
Re^2: Simple Coro + AnyEvent::HTTP example? by perrin (Chancellor) on Mar 19, 2009 at 19:25 UTC
Last time I tried that, it was really slow. Much slower than just forking and using LWP normally in each process.	[reply]
Re^2: Simple Coro + AnyEvent::HTTP example? by Anonymous Monk on Mar 19, 2009 at 15:17 UTC
Good catch and certainly an alternative. I would still like to look into Coro because it is more generic. Most of the hosts I want to query have a HTTP interface but some don't. Of course I could write a wrapper and perhaps I will but if I could learn how to properly use Coro I could manage the different interfaces in a generic way.	[reply]