how to adapt this to incrementing strings perl-style

See a a recent example where I showed exactly that, using tie with MCE::Shared (that example shows incrementing the value of keys in a hash; see the MCE::Shared doc for sharing a scalar via tie).

Alternatively, if you are using MCE::Shared's OO interface it provides sugar methods (shown here using MCE::Hobo for workers):

my $shared = MCE::Shared->scalar(0);

sub task {
    $shared->incrby(1);
}

MCE::Hobo->init( max_workers => 8, posix_exit  => 1 );
MCE::Hobo->create( \&task, $_ ) for 0 .. 41;
MCE::Hobo->wait_all;

END {
    print "The answer is $shared\n";
}
[download]

Hope this helps!

Thanks again but I couldn't get Hobo to do it. I'm just a humble pilgrim in the Holy Land of MCE. Thanks to the seemingly infinite amount of elaborate example code I have tried many things. As you will see below my cargo cult strategy simply replaces numbers with letters to see if it works. MCE::Candy caught my eye for preserving output order because my old method did that, even thought it's not necessary. I found the magic in MCE::Candy::out_iter_array:

#!/usr/bin/perl
use strict;
use warnings;
use MCE;
use MCE::Candy;

my $volume      = 26*26;
my $max_workers = 4;
my $chunk_size  = int $volume / $max_workers;
my @results;

my $mce = MCE->new(
   max_workers => $max_workers,
   chunk_size  => $chunk_size,
   gather => MCE::Candy::out_iter_array(\@results),
   user_func => sub {
      my ($mce, $chunk_ref, $chunk_id) = @_;
      my @output;
      foreach my $item (@{ $chunk_ref }) {
         push @output, $item++;
      }
      $mce->gather($chunk_id, @output);
   }
);

$mce->process([ 'aa' .. 'zz' ]);

print "$_, " for @results;
print scalar @results, "\n";
[download]

https://github.com/marioroy/mce-perl/blob/master/README.md

Greetings,

Some helpful tips for processing a large array.

Spawn workers early before creating or obtaining a large array to be used as input data. Dividing the work equally by the number of workers is not recommended for large data sets. A chunk_size value of 4000 or 8000 is fine for large arrays. It doesn't take much (chunk_size wise) for IPC to not become the bottleneck. Finally, workers persist after processing (re: $mce->process). Thus, shutdown workers when completed. This is done for you when the script terminates if omitted.

#!/usr/bin/perl
use strict;
use warnings;
use MCE;
use MCE::Candy;

my $volume      = 26*26;
my $max_workers = 4;
my $chunk_size  = int $volume / $max_workers / 16;
my @results;

my $mce = MCE->new(
   max_workers => $max_workers,
   chunk_size  => $chunk_size,
   gather => MCE::Candy::out_iter_array(\@results),
   user_func => sub {
      my ($mce, $chunk_ref, $chunk_id) = @_;
      my @output;
      foreach my $item (@{ $chunk_ref }) {
         push @output, $item++;
      }
      $mce->gather($chunk_id, @output);
   }
)->spawn;

$mce->process([ 'aa' .. 'zz' ]);
$mce->shutdown;

print "$_, " for @results;
print scalar @results, "\n";
[download]

Regards, Mario

incrementing strings perl-style

You mean, as in perlop, string

matches the pattern /^[a-zA-Z]*[0-9]*\z/, the increment is done as a string, preserving each character within its range, with carry

> You mean, as in perlop

Yes and I've been re-reading that page for weeks :-)

What an unusual and interesting task. What's the context? And string is huge enough for parallelization overhead to pay off?

I'm doing with words what mathematicians do with numbers. It's very similar to searching for prime numbers. These liguistic primes are defined by rules of grammar. So my data is not huge strings but huge sets of billions, and trillions, and hundreds of trillions of uniform sized strings. You nailed what I have to do by suggesting chunking the data to equal sizes guaranteed not to increment. Perfect for MCE! Thank you.

This one liner prints all my data, one string at a time:

perl -e '$a="a";$n=1;$s=time;print"SEC\tWORD\tITER\n";while(){print"\r
+",time-$s,"\t$a\t$n   ";$a++&&$n++}'
[download]

I'll post a root node with more detail once I get it ported over to MCE.