Update: Modified script. Completes in 1 second with Cygwin Perl v5.22.4 and less than 1 second on Unix.
Update: Modified list of modules to pre-load. On the Windows platform, run with Perl v5.26 or later for best results.
The following is a demonstration using MCE::Loop. The behaviour is similar to running a pool of workers. Workers request the manager process the next URL to process. The job_delay option gives each worker time to load any missing modules at runtime before another worker starts processing. The delay occurs one time by each worker.
use strict;
use warnings;
use LWP::Simple;
# Pre-load essential modules for extra stability.
if ( $INC{'LWP/UserAgent.pm'} && !$INC{'Net/HTTP.pm'} ) {
require IO::Handle;
require Net::HTTP;
require Net::HTTPS;
}
my @urls = (
'http://hooboy.no-such-host.int/',
'http://us.a1.yimg.com/us.yimg.com/i/ww/m5v9.gif',
'http://www.guardian.co.uk/',
'http://www.ora.com/ask_tim/graphics/asktim_header_main.gif',
'http://www.pixunlimited.co.uk/siteheaders/Guardian.gif',
'http://www.yahoo.com',
) x 4;
use MCE::Loop;
MCE::Loop->init(
max_workers => 8,
chunk_size => 1,
posix_exit => 1,
);
mce_loop {
my ($type, $length, $mod) = head( my $url = $_ );
print "$url is done\n";
} \@urls;
MCE::Loop->finish;
Regards, Mario
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.