Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have a script that loads an array of URLs once a day (one-a-day sale sites like Woot.com, shopping.yahoo.com). Sometimes it takes a few seconds longer for some URLs to load and I don't want to wait.

Can someone show me a quick example of how I could branch off and multi-process this to load more at once? 

my @urls = qq("http://www.woot.com", "http://www.shopping.yahoo.com");

my %stored_data;
foreach my $page (@urls)
{
   my $source = get($page);
   $stored_data{$page} = $source;
   print "loaded and saved $page\n";
}

[download]

Replies are listed 'Best First'.
Re: forking/threads/running more than one command at once
by BrowserUk (Patriarch) on Oct 05, 2007 at 22:20 UTC

    You could try something like this. 

    #! perl -slw
use strict;
use threads;
use threads::shared;;
use LWP::Simple;
use constant MAX_THREADS => 3;

chomp( my @urls = <DATA> );

my %store :shared;
my $threads :shared;
for my $page ( @urls ) {
    async {
        $store{ $page } = get( 'http://' . $page ) or die "Failed to g
+et $page";
        print "Loaded $page";
    } and $threads++;
    if( $threads > MAX_THREADS ) {
        if( my @joinable = threads->list(threads::joinable) ) {
            $_->join for @joinable;
            $threads -= @joinable;
        }
        else {
           sleep 1;
       }
    }
}
$_->join for threads->list;

printf "Press enter to see the data gathered"; <STDIN>;

print "*** $_ ***\n$store{ $_ }\n\n" for keys %store;

__END__
www.yahoo.com
www.microsoft.com
www.bbc.co.uk
www.ibm.com
www.google.com
www.bt.co.uk

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
Re: forking/threads/running more than one command at once
by ikegami (Patriarch) on Oct 05, 2007 at 20:55 UTC
[reply]
Re: forking/threads/running more than one command at once
by Corion (Patriarch) on Oct 05, 2007 at 21:16 UTC

    Also, Parallel::Iterator - but it's relatively new, so there likely are some bugs left in it.

[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.

Back to Seekers of Perl Wisdom