Re^5: Iteration through large array using a N number of forks.

I agree. The examples for LWP::Parallel are rather intimidating.

I think this is about as easy as parallel LWP gets:

#! perl -slw
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use LWP::Simple;
use Time::HiRes qw[ time ];

$|=1;
our $THREADS    ||= 3;
our $DELAY        ||= 10;

sub ping {
    my( $Q, $done ) = @_;
    my $tid = threads->self->tid;
    while( not $$done ) {
        my( $time, $url ) = split ':', $Q->dequeue;
        select undef, undef, undef, 0.01, while time < $time;
        my $start = time;
        printf "($tid) %20s returned [%50.50s] and took %f seconds\n"
            , $url
            , join( ' ', grep{ defined } head( "http://$url" ) )
            , time() - $start
        ;
        $Q->enqueue( ( $DELAY + time() ) . ":$url" );
    }
}

my $Q = new Thread::Queue;
$Q->enqueue( map{ chomp; time() . ":$_" } <DATA> );

my $done : shared = 0;
my @threads = map{ threads->new( \&ping, $Q, \$done ) } 1 .. $THREADS;

<STDIN>;
print "Stopping...";

$done = 1;
$_->join for @threads;

__DATA__
www.yahoo.com
www.aol.com
www.altavista.com
www.time.com
www.whitehouse.gov
www.parliament.uk
www.scottish.parliament.uk
www.europarl.org.uk
www.nasa.com
www.perl.com
www.perl.org
www.activestate.com
www.bbc.co.uk
www.ibm.com
www.google.com
www.cnn.com
www.perlmonks.com
www.microsoft.com
[download]

Output (truncated for posting)

P:\test>435950
(1)        www.yahoo.com returned [           text/html] and took 0.88
+7310 
(2)          www.aol.com returned [      OLserver/3.4.2] and took 0.79
+4034 
(3)    www.altavista.com returned [      9-1 1109797758] and took 0.76
+6369 
(3)    www.parliament.uk returned [      rosoft-IIS/5.0] and took 0.45
+3125 
(2)   www.whitehouse.gov returned [      9797455 Apache] and took 0.60
+9375 
(1)         www.time.com returned [    teEnterprise/4.1] and took 1.10
+9375 
(2)  www.europarl.org.uk returned [      rosoft-IIS/4.0] and took 0.46
+8750 
(3) www.scottish.parliament.uk returned [text/html Oracle9iAS] and too
+k 0.812500
(1)         www.nasa.com returned [      rosoft-IIS/5.0] and took 0.54
+6875 
(2)         www.perl.com returned [text/h/1.3.33 (Unix)] and took 0.78
+1250 
(1)  www.activestate.com returned [      rosoft-IIS/5.0] and took 0.54
+6875 
(2)        www.bbc.co.uk returned [      /2.0.51 (Unix)] and took 0.39
+0625 
(3)         www.perl.org returned [text/h7760 Apache/1.] and took 1.09
+3750 
(3)          www.cnn.com returned [      9797821 Apache] and took 0.56
+2500 
(1)          www.ibm.com returned [text/h26.3  Apache/1] and took 1.03
+1250 
(2)       www.google.com returned [      html 0 GWS/2.1] and took 1.35
+9375 
(1)    www.microsoft.com returned [text/h3 Microsoft-II] and took 0.62
+5000 
(3)    www.perlmonks.com returned [       Apache/1.3.27] and took 3.73
+4375 
(3)    www.altavista.com returned [      9-1 1109797769] and took 0.59
+3750 
(2)        www.yahoo.com returned [           text/html] and took 0.81
+2500 
(1)          www.aol.com returned [      OLserver/3.4.2] and took 0.81
+2500 
(3)    www.parliament.uk returned [      rosoft-IIS/5.0] and took 0.40
+6250 
(2)   www.whitehouse.gov returned [      9797455 Apache] and took 0.39
+0625 
(3)  www.europarl.org.uk returned [      rosoft-IIS/4.0] and took 0.39
+0625 
(3)         www.nasa.com returned [      rosoft-IIS/5.0] and took 0.59
+3750 
(1)         www.time.com returned [    teEnterprise/4.1] and took 1.21
+8750 
(2) www.scottish.parliament.uk returned [text/html Oracle9iAS] and too
+k 0.937500
(2)        www.bbc.co.uk returned [      /2.0.51 (Unix)] and took 0.43
+7500 
(3)         www.perl.com returned [text/h/1.3.33 (Unix)] and took 0.60
+9375 
(1)  www.activestate.com returned [      rosoft-IIS/5.0] and took 0.60
+9375 
(2)         www.perl.org returned [text/h7772 Apache/1.] and took 1.54
+6875 
(3)          www.cnn.com returned [      9797833 Apache] and took 1.73
+4375 
(3)    www.microsoft.com returned [text/h3 Microsoft-II] and took 0.45
+3125 
(1)          www.ibm.com returned [text/h26.3  Apache/1] and took 2.01
+5625 
(2)       www.google.com returned [      html 0 GWS/2.1] and took 1.37
+5000 

Stopping...
(3)    www.perlmonks.com returned [       Apache/1.3.27] and took 1.85
+9375 
(1)    www.altavista.com returned [      9-1 1109797779] and took 0.78
+1250 
(2)        www.yahoo.com returned [           text/html] and took 0.70
+3125
[download]

Examine what is said, not who speaks.

Silence betokens consent.

Love the truth but pardon error.

Comment on Re^5: Iteration through large array using a N number of forks. Select or Download Code