Hi PerlMonks! After years of using Parallel::ForkManager to speed up making thousands of HTTP request, I was adviced by Ikegami at Stackoverflow.com to give Net::Curl::Multi a try. And I have to say its awesome! Now, becouse its my first time using it, and becouse final version of the code will have to deal with external, 3rd party API, I would really want to be sure its working fine and doesnt have any bugs. Below You will find my test script based on Ikegami's advice. It works, I dont have any problems with it, but maybe some more experienced Perl users would notice something that I didnt, or tell me if I shouldn't do something like I did, etc. My main concern is the LINE loop and way that I add URL to N::C::M queue, fill the queue with more URLs, wait for one of them to finish, and go to the next URL and add it. Some notes:

• Im using external txt file with URLs becouse there might be A LOT of them (hundret of thousands even) so I dont want to load everything to array.
• As I don't fully understand how Net::Curl::Multi's concurrency works, please tell me whether I should expect any problems with putting MySQL UPDATE command (via DBI) inside RESPONSE loop (besides higher server load obviously - I expect final script to run with at least 50 concurrent N::C::M workers)

Thanks!

#!/usr/bin/perl

use Net::Curl::Easy  qw( :constants );
use Net::Curl::Multi qw( );

sub make_request {
    my ( $url ) = @_;
    my $easy = Net::Curl::Easy->new();
    $easy->{url} = $url;
    $easy->setopt( CURLOPT_URL,        $url );
    $easy->setopt( CURLOPT_HEADERDATA, \$easy->{head} );
    $easy->setopt( CURLOPT_FILE,       \$easy->{body} );
    return $easy;
}

my $maxWorkers = 10;

my $multi = Net::Curl::Multi->new();
my $workers = 0;

my $i = 1;
open my $fh, "<", "urls.txt";
LINE: while ( my $url = <$fh> )
{
    chomp( $url );
    $url .= "?$i";
    print "($i) $url\n";
    my $easy = make_request( $url );
    $multi->add_handle( $easy );
    $workers++;

    my $running = 0;
    do {
        my ($r, $w, $e) = $multi->fdset();
        my $timeout = $multi->timeout();
        select $r, $w, $e, $timeout / 1000
        if $timeout > 0;

        $running = $multi->perform();
        RESPONSE: while ( my ( $msg, $easy, $result ) = $multi->info_r
+ead() ) {
            $multi->remove_handle( $easy );
            $workers--;
            printf( "%s getting %s\n", $easy->getinfo( CURLINFO_RESPON
+SE_CODE ), $easy->{url} );
        }

        # dont max CPU while waiting
        select( undef, undef, undef, 0.01 );
    } while ( $workers == $maxWorkers || ( eof && $running ) );
    $i++;
}
close $fh;
[download]