in reply to Re^2: Prima + MCE::Hobo demonstration
in thread Prima + MCE::Hobo demonstration

For completeness, here's the code using Parallel::ForkManager. Calling MCE::Shared->init() is important to spread out IPC across 12 data channels. Not doing so, means all 200 workers share a single data channel and take much longer to run. Init is called automatically for MCE, MCE::Hobo and threads.

Parallel::ForkManager

use strict;
use warnings;

use Parallel::ForkManager;
use MCE::Shared;
use Time::HiRes 'time';

my $n = MCE::Shared->scalar(0);
my $s = time;

my $fork_manager = new Parallel::ForkManager(200);

$fork_manager->set_waitpid_blocking_sleep(0);

foreach my $child ( 1 .. 200 ) {
   my $pid = $fork_manager->start($child) and next;

   # init optionally takes an integer value
   MCE::Shared->init($child);

   my $v; $v = $n->incr() for 1 .. 933;

   $fork_manager->finish($child);
}

$fork_manager->wait_all_children;

printf "duration: %0.3f\n", time - $s;
print $n->get(), "\n";

__END__

main process, shared-manager process, and 200 workers

Parallel::ForkManager: ~ 1200K per worker

[download]

It's great knowing that MCE::Hobo isn't far behind regarding memory consumption. What I've learn is that simply loading modules doesn't tell the whole story. One must capture memory consumption while running for a better picture. :)

Regards, Mario

Replies are listed 'Best First'.
Re^4: Prima + MCE::Hobo demonstration
by zentara (Cardinal) on May 02, 2017 at 13:56 UTC
    Hello mariojoy! I have one question I've been meaning to ask you about MCE::Shared, which setups up some IPC with filehandles passed thru a socketpair? Is this correct? I read in the docs about MCE using IO::FDPass. Can you elaborate on why you pass a filedescriptor, rather than just using the socketpair for communication? Can you elaborate on how this works, and any pitfalls it may have?
    I'm not really a human, but I play one on earth. ..... an animated JAPH
[reply]

      Hi zentara,

      For MCE::Shared, the use of IO::FDPass applies to Condvar, Queue and to mce_open when constructing a shared handle from a non-shared handle not yet knowed to the shared-manager process. IO::FDpass isn't used for anything else.

         use MCE::Shared;

   # starting the shared-manager, implicitly
   my $ca = MCE::Shared->cache( max_keys => 500 );

   # or starting early, explicitly
   MCE::Shared->start();

   # at this point, IO::FDPass is necessary for constructing
   # a shared condvar or queue, passing fd descriptor

   # ... for new socket handle made during construction
   # workers block using a socket handle for ->wait, ->timedwait
   # the shared-manager unblocks by writing to _cw_sock

   my $cv = MCE::Shared->condvar();  # _cw_sock

   # ... for new socket handles made during construction
   # workers block using socket handles for ->dequeue, ->await
   # the shared-manager unblocks by writing to _qw_sock, _aw_sock

   my $q1 = MCE::Shared->queue();  # _qw_sock
   my $q2 = MCE::Shared->queue( await => 1 );  # _aw_sock

      [download]

      Without IO::FDPass, one cannot construct a shared condvar or queue while the shared manager is running. The work around is to construct these before other shared objects (e.g. ->array, ->hash, etc). Then, start the shared-manager manually.

         use MCE::Shared;

   my $cv  = MCE::Shared->condvar(0);
   my $que = MCE::Shared->queue( fast => 1 );

   MCE::Shared->start() # must start manually

   my $hash = MCE::Shared->hash(); # or implicitly

      [download]

      Note: MCE starts the shared-manager if not started. Ditto for MCE::Hobo because that constructs two shared hashes.

      Regarding mce_open, IO::FDPass is needed when constructing a shared-handle from a non-shared handle not yet available inside the shared-manager process. The workaround is having the non-shared handle made before the shared-manager process is started implicitly or explicitly.

         # the shared-manager knows of \*STDIN, \*STDOUT, \*STDERR

   mce_open my $shared_in,  "<",  \*STDIN;   # ok
   mce_open my $shared_out, ">>", \*STDOUT;  # ok
   mce_open my $shared_err, ">>", \*STDERR;  # ok

   # but may not know of $non_shared_fh

   mce_open my $shared_fh, ">>", $non_shared_fh;

      [download]

      Regards, Mario

[reply]
[d/l]
[select]

In Section Cool Uses for Perl