Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to get Perl to use all CPU cores with MCE. I have read and tried some examples but can't seem to grasp it. Can anyone explain MCE for dummies? I was hoping this would produce a large number but it only prints 0: 
#!/usr/bin/perl

use strict;
use warnings;
use MCE::Loop;

our $fu = 0;

MCE::Loop::init {
   max_workers => 8,
   chunk_size => 1000
};

mce_loop { do_work($_) } 1..10000000;
print "Done: $fu\n";

sub do_work { $fu += $_[0] }

[download]

Replies are listed 'Best First'.
Re: Getting started with MCE (the Many-Core Engine)
by vr (Curate) on Jun 11, 2018 at 08:49 UTC 
    use strict;
use warnings;
use MCE;
use MCE::Loop;
use MCE::Shared;

my $volume      = 1e9;          # crank it up
my $max_workers = MCE::Util::get_ncpu;
my $chunk_size  = int $volume / $max_workers;

MCE::Loop::init { 
    max_workers => $max_workers,
    chunk_size  => $chunk_size,
    bounds_only => 1,
};

my $fu = MCE::Shared-> scalar( 0 );

mce_loop_s {                    # note "s"
    my $partial_result  = 0;
    my ( $begin, $end ) = @$_;
    $partial_result    += $_ for $begin .. $end;

    print 'Worker', MCE-> wid, ' here, my partial result is ',
          $partial_result, "\n";

    $fu-> incrby( $partial_result );
} 1, $volume;

print "\nDone: @{[ $fu-> get ]}\n";

__END__

Worker4 here, my partial result is 218750000125000000
Worker1 here, my partial result is 31250000125000000
Worker3 here, my partial result is 156250000125000000
Worker2 here, my partial result is 93750000125000000

Done: 500000000500000000

    [download]

    Hi, not sure if it explains for dummies or not. Feel free to ask further, if it doesn't. Important:

    • it's required to provide way to communicate results back from workers which are just separate processes (or threads) -- i.e. to arrange for IPC. You didn't, thus your variable remained zero. One possibility, through explicitly shared variable, is shown above;
    • you really don't want to build an enormous list (1e7 elements in your code, or moreover 1e9 above) before any work is even started. For sequences, the MCE provides built-in methods. For other kind of input, think about e.g. setting up iterator as source.

[reply]
[d/l]
      Eureka! Adding a billion numbers was taking the usual single perl process 26.8 seconds but MCE spawns 8 perls (1 process per CPU core with get_ncpu), giving the fan a nice workout and doing it in 7.3 seconds! 365% faster FTW! 
      time perl -le '$x=1e9;$y=0;$y+=$_ for 1..$x;print$y'
500000000500000000

real    0m26.796s
user    0m26.617s
sys     0m0.088s

time mce (your script)
500000000500000000

real    0m7.323s
user    0m55.095s
sys     0m0.127s

      [download]
      Thank you so much for the example code. I can now see the meaning of chunk_size and how to share data between processes. MCE is a really beautiful distribution with comprehensive documentation and tons of examples (and a couple of cookbooks). I'm just not familiar with the concepts and jargon of parallel processing and was feeling kinda lost, but less now. Thanks to you! :-)

      Time for another go at the docs while I figure out how to adapt this to incrementing strings perl-style: 

      perl -le '$a="a";for(1..30){print$a++}'

      [download]
[reply]
[d/l]
[select]

        how to adapt this to incrementing strings perl-style

        See a a recent example where I showed exactly that, using tie with MCE::Shared (that example shows incrementing the value of keys in a hash; see the MCE::Shared doc for sharing a scalar via tie).

        Alternatively, if you are using MCE::Shared's OO interface it provides sugar methods (shown here using MCE::Hobo for workers): 

        my $shared = MCE::Shared->scalar(0);

sub task {
    $shared->incrby(1);
}

MCE::Hobo->init( max_workers => 8, posix_exit  => 1 );
MCE::Hobo->create( \&task, $_ ) for 0 .. 41;
MCE::Hobo->wait_all;

END {
    print "The answer is $shared\n";
}

        [download]

        Hope this helps!


        The way forward always starts with a minimal test.
[reply]
[d/l]
[select]
        incrementing strings perl-style

        You mean, as in perlop, string

        matches the pattern /^[a-zA-Z]*[0-9]*\z/, the increment is done as a string, preserving each character within its range, with carry
        ? What an unusual and interesting task. What's the context? And string is huge enough for parallelization overhead to pay off? Off the top of my head, split the string into more or less equal substrings (their number equal to number of workers), guaranteed not to carry when incremented required number of times -- or, find indices within original, then feed the data to MCE::Map, then concatenate the output :-)
[reply]
[d/l]
[select]
        Yes I did notice that 8 perl processes vs 1 "only" sped things up about 3.5x. Tests show no gains over 4 workers here. I guess these i7 processors have 4 physical cores virtualized to 8 marketing thingies. Without MCE my computationally intensive code seemed to be using the resources of 1/2 of 1 of the four real cores (1 virtual core? teenage CPU %s). MCE appears to unlock the missing ~85% of my CPU power. It's like I downloaded 3.5 more of these 4k computers from CPAN for FREE with MCE (and 1 post++ from Perlmonk vr): A $14,000 value! Unreal...
[reply]

Back to Seekers of Perl Wisdom