Hello karlgoethebier,

The demonstration looks fine. Some modules may not play well with threads, unfortunately. The LWP::Simple module has many dependencies. One of them may be unsafe for use with threads. In that case, the use_threads option is necessary to have workers spawn via fork on the Windows platform or when loading threads at the top of the script.

Network related tasks may benefit from MCE's interval option. It helps stagger the immediate code that follows. For this use-case, calling yield prevents workers from initiating remote connections at the same time. It is similarly to sleep, but runs serially, not parallel, for that duration of time. All participating workers wait their turn to sleep.

The next MCE update 1.830 will default to 1 for the posix_exit option. It is nearly impossible to manage a list of modules thread-safe or multi-process END safe for that matter.

#!/usr/bin/env perl

# http://www.perlmonks.org/?node_id=1192821

use strict;
use warnings;
use MCE::Loop;
use MCE::Shared;
use LWP::Simple;
use feature qw(say);

my $result = MCE::Shared->hash;
my @urls   = qw(http://perlmonks.org http://www.whitehouse.org);

MCE::Loop::init {
    max_workers => 'auto',
    chunk_size  => 1,
    interval    => 0.008,
    posix_exit  => 1,
    use_threads => 0
};

my $fetch = sub {
    eval { head(shift) };
    warn $@ if $@;
};

mce_loop {
    MCE->yield;
    my @data = $fetch->( $_ );
    $result->set( $_ => \@data );

}
@urls;

{
    no warnings qw(uninitialized);
    my $iter = $result->iterator();

    while ( my ( $url, $data ) = $iter->() ) {
        say $url;
        say for @$data;
        say q(---);
    }
}
[download]

MCE Loop is wantarray-aware. This allows one to use the gather method to send the key-value pair into a plain hash. For readers, this is how it was done before MCE::Shared came about.

#!/usr/bin/env perl

# http://www.perlmonks.org/?node_id=1192821

use strict;
use warnings;
use MCE::Loop;
use LWP::Simple;
use feature qw(say);

my @urls = qw(http://perlmonks.org http://www.whitehouse.org);

MCE::Loop::init {
    max_workers => 'auto',
    chunk_size  => 1,
    interval    => 0.008,
    posix_exit  => 1,
    use_threads => 0
};

my $fetch = sub {
    eval { head(shift) };
    warn $@ if $@;
};

my %result = mce_loop {
    MCE->yield;
    my @data = $fetch->( $_ );
    MCE->gather( $_ => \@data );

}
@urls;

{
    no warnings qw(uninitialized);

    while ( my ( $url, $data ) = each %result ) {
        say $url;
        say for @$data;
        say q(---);
    }
}
[download]

Regards, Mario

Workers yielding serially matters more when involving an event loop. For example preventing 200 workers x 300 chunk size from initiating many connections simultaneously.

 sub walk {
     my ( $job, $result, $failed ) = @_;

     # Yielding is critical when running an event loop in parallel.
     # Not doing so means that the app may reach contention points
     # with the firewall and likely impose unnecessary hardship at
     # the OS level. The idea here is not to have multiple workers
     # initiate HTTP requests to a batch of URLs at the same time.
     # Yielding in 1.827+ behaves more like scatter for the worker
     # to run solo in a fraction of time.

     MCE::Hobo->yield( 0.03 );   # MCE::Hobo 1.827

     my $cv = AnyEvent->condvar();

     # Populate the hash ref for URLs it could reach.
     # Do not mix AnyEvent timeout and Hobo timeout.
     # Choose to do the event timeout if available.

     foreach my $url ( @{ $job->{INPUT} } ) {
         $cv->begin();

         http_get $url, timeout => 2, sub {
             my ( $data, $headers ) = @_;
             $result->{$url} = $data;
             $cv->end();
         };
     }

     $cv->recv();

     # Populate the array ref for URLs it could not reach.

     foreach my $url ( @{ $job->{INPUT} } ) {
         push @{ $failed }, $url unless (exists $result->{ $url });
     }

     return;
 }
[download]

Regards, Mario

Thank you very much Mario and best regards, Karl

«The Crux of the Biscuit is the Apostrophe»

Furthermore I consider that Donald Trump must be impeached as soon as possible

C:\>perl -le "rubbish 'foo';"
String found where operator expected at -e line 1, near "rubbish 'foo'
+"
        (Do you need to predeclare rubbish?)
syntax error at -e line 1, near "rubbish 'foo'"
Execution of -e aborted due to compilation errors.
[download]

Thanks. I understood this so far. I had no better idea for the subject line.

Best regards, Karl

«The Crux of the Biscuit is the Apostrophe»

Furthermore I consider that Donald Trump must be impeached as soon as possible

Hi Karl, I am not clever enough to know how this could lead to the warning, but do you need to use a mutex since you are writing to a shared data structure?

...
use MCE::Loop;
use MCE::Shared;
use MCE::Mutex;

my $result = MCE::Shared->hash;
my $mutex = MCE::Mutex->new
...

mce_loop {
    my @data = $fetch->($_);
    sleep AMOUNT;
    $mutex->enter( $result->set( $_ => \@data ) );
}
@urls;

...
[download]

Hope this helps! (Mario would know better...)

Hi 1nickt. Yes, a mutex is necessary when workers update the same element in a shared hash. For Karl's demonstration, a mutex is optional when workers update unique elements via the OO interface.

Thanks for clarifying. I hadn't made the distinction, but I see it's an important one, certainly don't want to block if not needed.



The way forward always starts with a minimal test.

According to Mario i don't need a mutex - as far as i remember ;-)

Update: I tried it with the mutex as you suggested. Doesn't help.

Thanks and best regards, Karl

«The Crux of the Biscuit is the Apostrophe»

Furthermore I consider that Donald Trump must be impeached as soon as possible

Hi! Add next line my script - fix this error for me(Do you need to predeclare croak):

 use IO::Handle;