in reply to Re^2: Getting started with MCE (the Many-Core Engine)
in thread Getting started with MCE (the Many-Core Engine)

how to adapt this to incrementing strings perl-style

See a a recent example where I showed exactly that, using tie with MCE::Shared (that example shows incrementing the value of keys in a hash; see the MCE::Shared doc for sharing a scalar via tie).

Alternatively, if you are using MCE::Shared's OO interface it provides sugar methods (shown here using MCE::Hobo for workers): 

my $shared = MCE::Shared->scalar(0);

sub task {
    $shared->incrby(1);
}

MCE::Hobo->init( max_workers => 8, posix_exit  => 1 );
MCE::Hobo->create( \&task, $_ ) for 0 .. 41;
MCE::Hobo->wait_all;

END {
    print "The answer is $shared\n";
}

[download]

Hope this helps!


The way forward always starts with a minimal test.

Replies are listed 'Best First'.
Re^4: Getting started with MCE (the Many-Core Engine)
by Anonymous Monk on Jun 13, 2018 at 00:18 UTC
    Thanks again but I couldn't get Hobo to do it. I'm just a humble pilgrim in the Holy Land of MCE. Thanks to the seemingly infinite amount of elaborate example code I have tried many things. As you will see below my cargo cult strategy simply replaces numbers with letters to see if it works. MCE::Candy caught my eye for preserving output order because my old method did that, even thought it's not necessary. I found the magic in MCE::Candy::out_iter_array: 
    #!/usr/bin/perl
use strict;
use warnings;
use MCE;
use MCE::Candy;

my $volume      = 26*26;
my $max_workers = 4;
my $chunk_size  = int $volume / $max_workers;
my @results;

my $mce = MCE->new(
   max_workers => $max_workers,
   chunk_size  => $chunk_size,
   gather => MCE::Candy::out_iter_array(\@results),
   user_func => sub {
      my ($mce, $chunk_ref, $chunk_id) = @_;
      my @output;
      foreach my $item (@{ $chunk_ref }) {
         push @output, $item++;
      }
      $mce->gather($chunk_id, @output);
   }
);

$mce->process([ 'aa' .. 'zz' ]);

print "$_, " for @results;
print scalar @results, "\n";

    [download]
    https://github.com/marioroy/mce-perl/blob/master/README.md
[reply]
[d/l]

      Greetings,

      Some helpful tips for processing a large array.

      Spawn workers early before creating or obtaining a large array to be used as input data. Dividing the work equally by the number of workers is not recommended for large data sets. A chunk_size value of 4000 or 8000 is fine for large arrays. It doesn't take much (chunk_size wise) for IPC to not become the bottleneck. Finally, workers persist after processing (re: $mce->process). Thus, shutdown workers when completed. This is done for you when the script terminates if omitted.

      #!/usr/bin/perl
use strict;
use warnings;
use MCE;
use MCE::Candy;

my $volume      = 26*26;
my $max_workers = 4;
my $chunk_size  = int $volume / $max_workers / 16;
my @results;

my $mce = MCE->new(
   max_workers => $max_workers,
   chunk_size  => $chunk_size,
   gather => MCE::Candy::out_iter_array(\@results),
   user_func => sub {
      my ($mce, $chunk_ref, $chunk_id) = @_;
      my @output;
      foreach my $item (@{ $chunk_ref }) {
         push @output, $item++;
      }
      $mce->gather($chunk_id, @output);
   }
)->spawn;

$mce->process([ 'aa' .. 'zz' ]);
$mce->shutdown;

print "$_, " for @results;
print scalar @results, "\n";

      [download]

      Regards, Mario

[reply]
[d/l]
        Some helpful tips for processing a large array.

        Greetings Mario. What a marvelous coincidence. I was just writing a question on both topics you address. First, thank you for the free software! MCE is amazing.

        I've been tweaking it and noticed that smaller chunks are faster. For some reason 11k is the fastest here. I also noticed some delay in shutdown due to not calling shutdown but didn't know what was wrong. Thanks for the clue. Another issue I've come across is the value of MCE::Util::get_ncpu. On my machine (i7) it says 8 and of course it spawns 8 helpers but for some reason it's faster when I set that to 4: 

        > time mce

volume:         11881376
chunk_size:     16000
max_workers:    8

real    0m19.776s
user    0m43.350s
sys     0m10.321s

> time mce

volume:         11881376
chunk_size:     16000
max_workers:    4

real    0m17.615s
user    0m27.495s
sys     0m4.836s

        [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom