perlfan1020 has asked for the wisdom of the Perl Monks concerning the following question:

I need to resolve a large number of HTTP requests asynchronously and write the retrieved HTTP pages to files named from the URLs.

AnyEvent::HTTP seems to be a good choice. However, in the call back as the following: I can't associate the $data to the $url.

http_get $url, key => value..., $cb->($data, $headers)

Any ideas? Thanks in advance!

Replies are listed 'Best First'.
Re: Asynchronous HTTP requests
by Loops (Curate) on Nov 23, 2014 at 06:01 UTC

    It's in the header... 

    use AnyEvent::Loop;
use AnyEvent::HTTP;
http_get 'http://yahoo.com', sub { my ($body, $hdr) = @_; die $hdr->{U
+RL} };
AnyEvent::Loop::run;

    [download]

    And if it wasn't, or there is other information you'd like to use.. capture it as a closure when the sub is defined.. for instance:

    use AnyEvent::Loop;
use AnyEvent::HTTP;
use feature 'say';
for my $url ('yahoo.com', 'google.com') {
        http_get "http://$url", sub { say STDERR $url };
}
AnyEvent::Loop::run;

    [download]

    Two separate callback subs are created, and in each of them $url will be set to the value it held when the http_get was run.

[reply]
[d/l]
[select]
Re: Asynchronous HTTP requests
by ikegami (Patriarch) on Nov 25, 2014 at 19:17 UTC
[reply]

Back to Seekers of Perl Wisdom