Greetings,

I reached my goal in reaching below 2M per worker. Testing involved running 200 workers with and without threads for the MCE demonstration and without threads for MCE::Hobo.

MCE::Flow

use strict;
use warnings;

use threads;  # comment out for child processes

use MCE::Flow;
use MCE::Shared;
use Time::HiRes 'time';

my $n = MCE::Shared->scalar(0);
my $s = time;

sub task {
   # do something, plenty time to see top
   my $v; $v = $n->incr() for 1 .. 933;
}

mce_flow { max_workers => 200 }, \&task;

MCE::Flow::finish;

printf "duration: %0.3f\n", time - $s;
print $n->get(), "\n";

__END__

main process, shared-manager process, and 200 workers

# with threads: e.g. use threads

498M  MCE 1.827, MCE::Shared 1.824
437M  MCE 1.828, MCE::Shared 1.825  <- big reduction
399M  MCE 1.829, MCE::Shared 1.826  <- more reduction

# without threads

MCE 1.829, MCE::Shared 1.826: ~ 1744K per worker
[download]

MCE::Hobo

use strict;
use warnings;

use MCE::Hobo;
use MCE::Shared;
use Time::HiRes 'time';

my $n = MCE::Shared->scalar(0);
my $s = time;

sub task {
   # do something, plenty time to see top
   my $v; $v = $n->incr() for 1 .. 933;
}

MCE::Hobo->create(\&task) for 1 .. 200;
MCE::Hobo->waitall;

printf "duration: %0.3f\n", time - $s;
print $n->get(), "\n";

__END__

main process, shared-manager process, and 200 workers

MCE::Shared 1.826: ~ 1356K per worker
[download]

Well, I never imagined for MCE::Hobo to consume less than 1.4M per worker, let alone MCE::Flow running below 2M. MCE 1.829 and MCE::Shared 1.826 will be released later this week.

Regards, Mario

For completeness, here's the code using Parallel::ForkManager. Calling MCE::Shared->init() is important to spread out IPC across 12 data channels. Not doing so, means all 200 workers share a single data channel and take much longer to run. Init is called automatically for MCE, MCE::Hobo and threads.

Parallel::ForkManager

use strict;
use warnings;

use Parallel::ForkManager;
use MCE::Shared;
use Time::HiRes 'time';

my $n = MCE::Shared->scalar(0);
my $s = time;

my $fork_manager = new Parallel::ForkManager(200);

$fork_manager->set_waitpid_blocking_sleep(0);

foreach my $child ( 1 .. 200 ) {
   my $pid = $fork_manager->start($child) and next;

   # init optionally takes an integer value
   MCE::Shared->init($child);

   my $v; $v = $n->incr() for 1 .. 933;

   $fork_manager->finish($child);
}

$fork_manager->wait_all_children;

printf "duration: %0.3f\n", time - $s;
print $n->get(), "\n";

__END__

main process, shared-manager process, and 200 workers

Parallel::ForkManager: ~ 1200K per worker
[download]

It's great knowing that MCE::Hobo isn't far behind regarding memory consumption. What I've learn is that simply loading modules doesn't tell the whole story. One must capture memory consumption while running for a better picture. :)

Regards, Mario

Hello mariojoy! I have one question I've been meaning to ask you about MCE::Shared, which setups up some IPC with filehandles passed thru a socketpair? Is this correct? I read in the docs about MCE using IO::FDPass. Can you elaborate on why you pass a filedescriptor, rather than just using the socketpair for communication? Can you elaborate on how this works, and any pitfalls it may have?

---

I'm not really a human, but I play one on earth. ..... an animated JAPH

MCE 1.829 and MCE::Shared 1.826 have been released. These include the updates for reducing memory consumption, especially for MCE::Shared::Server.