The following provides a demonstration. Notice how workers populate a local array before sending to the manager process using one gather call. This is one way to minimize IPC overhead.

use strict;
use warnings;

use MCE::Loop;
use MCE::Candy;

my $count  = 0;
my $input  = [ ];
my $output = [ ];

my $sample = { a => ++$count, b => ++$count, c => ++$count };

push @{ $input }, $sample for ( 1 .. 100000 );

MCE::Loop::init(
   max_workers => 3,
   chunk_size  => 100,
   gather      => MCE::Candy::out_iter_array( $output )
);

mce_loop {
   my ( $mce, $chunk_ref, $chunk_id ) = @_;
   my @ret;

   for my $h ( @{ $chunk_ref } ) {
      push @ret, {
         a => $h->{a} * 2,
         b => $h->{b} * 2,
         c => $h->{c} * 2,
      };
   }

   MCE->gather($chunk_id, @ret);

} $input;

MCE::Loop::finish();

print scalar(@{ $output }), "\n";
[download]

Regards, Mario.

Thanks, Mario. After implementing this suggestion I get the following with my real data:

Benchmark: timing 5 iterations of MCE loop             , MCE loop/batc
+h gather, MCE map              , Sequential loop      ...
MCE loop             : 56 wallclock secs ( 1.26 usr  0.57 sys + 36.11 
+cusr 42.06 csys = 80.00 CPU) @  0.06/s (n=5)
MCE loop/batch gather: 57 wallclock secs ( 0.55 usr  0.17 sys + 36.20 
+cusr 40.75 csys = 77.67 CPU) @  0.06/s (n=5)
MCE map              : 66 wallclock secs ( 3.75 usr  0.32 sys + 36.34 
+cusr 40.67 csys = 81.08 CPU) @  0.06/s (n=5)
Sequential loop      : 73 wallclock secs (35.16 usr + 28.66 sys = 63.8
+2 CPU) @  0.08/s (n=5)
[download]

Surprisingly the batching didn't seem to help. Going to keep working with it. Also thanks for your PMs.

The way forward always starts with a minimal test.

I concur with Mario, benchmarking in any virtualised environment is an almost meaningless act; the hypervisor will allocate your instance as much resource as it uses commensurate with: a) being able to allocate sufficient to all the other instances sharing the physical cpus; b) within the limits of the instance IO restrictions.

And remember, all cloud providers will over-provision physical cpus many times. And it gets worse, sometimes, if a "new" physical CPU has been fired up to run your newly started instance, then you may find yourself the only occupier for a brief while and get performance 10 or 20 times better than a few seconds or minutes later, once the grand overseer has noticed that there is an under utilised box running and transfers in a dozen other instances to soak up the spare capacity.

The only way I've found is to benchmark on standalone hardware for minimal resource -- cpu, io and resource -- not time; and the only meaningful measure of success is the billed cost of running the same job (on the same instance type) before and after optimisation.

And the calculations aren't easy, because the only real control you have is the size of the instance you purchase. There is no point in purchasing a huge compute instance, if your code's performance is bound by the IO limit for that instance. You are just paying more the same throughput.

And your purchase is made in terms of EC2 compute units, and Amazon isn't going to tell you how they are calculated.

Finally, MCE trades IO and CPU for time. On (your own) physical hardware that is a perfectly sane and sensible trade-off. IO costs you nothing, a 90% idle CPU is a waste. Your time is the only factor to consider.

But AWS instances bill you for IO and CPU and time. If for example, using MCE pushed your job into needing a bigger instance that allows higher IO, that could double or treble your costs! It is often better to run 10 tiny instances than one XL instance; if your task lends itself to that.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.

Testing and benchmarking on an AWS instance is more fun when the AWS instance has exclusive use of the CPU cores allocated to it. Chunking in MCE is reasonably fast. Perhaps, the physical box is over saturated with AWS instances. I'm hoping that the latter is not the case.

Regards, Mario.