MCE 1.881 update

I read code recently, using MCE::Map with 4 workers. But, only 2 workers were running. The update is a tiny edit to the _parse_chunk_size function.

1.881 Thu Oct 13 23:45:00 EST 2022

  * Improved the private _parse_chunk_size function. For better
    utilization of CPU cores in MCE::Grep, MCE::Map, and MCE::Stream,
    processing small input sizes.

    Previously, chunk_size => 'auto' equals 2 minimally.
    Starting with MCE v1.881, 'auto' equals 1 minimally.
[download]

It has been a while and forgotten what the overhead is running mce_map. Testing was done on a Linux box. Also, Perl has Sereal::Encoder and Sereal::Decoder installed.

native map

use strict;
use warnings;
use Time::HiRes 'time';

my $start = time();
my @ret = map { $_ * 2 } 1..10_000_000;

printf "%s %s\n", $ret[5_000_000], $ret[-1];
printf "%0.3f seconds\n", time() - $start;

__END__
10000002 20000000
0.561 seconds
[download]

mce_map

Workers request the next chunk from the manager process. The overhead is low, considering mce_map involves bidirectional data transfers (from manager, to manager).

use strict;
use warnings;
use MCE::Map;
use Time::HiRes 'time';

MCE::Map->init(
    chunk_size  => 'auto',
    max_workers => 8,
    posix_exit  => 1,
);

my $start = time();
my @ret = mce_map { $_ * 2 } 1..10_000_000;

printf "%s %s\n", $ret[5_000_000], $ret[-1];
printf "%0.3f seconds\n", time() - $start;

__END__
10000002 20000000
0.996 seconds
[download]

mce_map_s

For a sequence of numbers, append the suffix "_s" to "mce_map". Workers do not involve the manager process for receiving input. Instead, workers communicate the next starting number among themselves. The syntax is slightly different. Specify the starting and ending number after the code block (two args, notice the comma). Data transfer is unidirectional (to manager).

use strict;
use warnings;
use MCE::Map;
use Time::HiRes 'time';

MCE::Map->init(
    chunk_size  => 'auto',
    max_workers => 8,
    posix_exit  => 1,
);

my $start = time();
my @ret = mce_map_s { $_ * 2 } 1, 10_000_000;

printf "%s %s\n", $ret[5_000_000], $ret[-1];
printf "%0.3f seconds\n", time() - $start;

__END__
10000002 20000000
0.655 seconds
[download]

something different

I have seen the following MCE usage. Here, chunk_size is explicitly set to 1 behind the scene. One may omit the first two lines inside the code block.

use strict;
use warnings;
use Data::Dumper;
use MCE;
use Time::HiRes 'sleep';

my %res = ();
my $mce = MCE->new(
    max_workers => 5,
    posix_exit  => 1,
    gather      => \%res,
);

my @input = 1..10;

$mce->foreach(\@input, sub {
  # my ($mce, $chunk_ref, $chunk_id) = @_;
  # my $item = $chunk_ref->[0];   # or $_
    my $item = $_;

    MCE->printf("Worker %s began item $item\n", MCE->wid);
    sleep 0.01; # simulate work
    MCE->gather($item => $item * 2);

    MCE->printf("Worker %s ended item $item\n", MCE->wid);
});

$mce->shutdown;

print Dumper(\%res);
[download]

output

Worker 5 began item 3
Worker 4 began item 2
Worker 1 began item 1
Worker 2 began item 4
Worker 3 began item 5
Worker 5 ended item 3
Worker 4 ended item 2
Worker 1 ended item 1
Worker 5 began item 6
Worker 3 ended item 5
Worker 2 ended item 4
Worker 1 began item 8
Worker 4 began item 7
Worker 2 began item 9
Worker 3 began item 10 
Worker 5 ended item 6
Worker 1 ended item 8
Worker 4 ended item 7
Worker 3 ended item 10 
Worker 2 ended item 9
$VAR1 = {
          '9' => 18,
          '7' => 14,
          '3' => 6,
          '1' => 2,
          '5' => 10,
          '8' => 16,
          '4' => 8,
          '2' => 4,
          '10' => 20,
          '6' => 12
        };
[download]