Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^2: Prima + MCE::Hobo demonstration

by marioroy (Prior)
on May 02, 2017 at 07:19 UTC ( [id://1189307]=note: print w/replies, xml ) Need Help??


in reply to Re: Prima + MCE::Hobo demonstration
in thread Prima + MCE::Hobo demonstration

Greetings,

I reached my goal in reaching below 2M per worker. Testing involved running 200 workers with and without threads for the MCE demonstration and without threads for MCE::Hobo.

MCE::Flow

use strict; use warnings; use threads; # comment out for child processes use MCE::Flow; use MCE::Shared; use Time::HiRes 'time'; my $n = MCE::Shared->scalar(0); my $s = time; sub task { # do something, plenty time to see top my $v; $v = $n->incr() for 1 .. 933; } mce_flow { max_workers => 200 }, \&task; MCE::Flow::finish; printf "duration: %0.3f\n", time - $s; print $n->get(), "\n"; __END__ main process, shared-manager process, and 200 workers # with threads: e.g. use threads 498M MCE 1.827, MCE::Shared 1.824 437M MCE 1.828, MCE::Shared 1.825 <- big reduction 399M MCE 1.829, MCE::Shared 1.826 <- more reduction # without threads MCE 1.829, MCE::Shared 1.826: ~ 1744K per worker

MCE::Hobo

use strict; use warnings; use MCE::Hobo; use MCE::Shared; use Time::HiRes 'time'; my $n = MCE::Shared->scalar(0); my $s = time; sub task { # do something, plenty time to see top my $v; $v = $n->incr() for 1 .. 933; } MCE::Hobo->create(\&task) for 1 .. 200; MCE::Hobo->waitall; printf "duration: %0.3f\n", time - $s; print $n->get(), "\n"; __END__ main process, shared-manager process, and 200 workers MCE::Shared 1.826: ~ 1356K per worker

Well, I never imagined for MCE::Hobo to consume less than 1.4M per worker, let alone MCE::Flow running below 2M. MCE 1.829 and MCE::Shared 1.826 will be released later this week.

Regards, Mario

Replies are listed 'Best First'.
Re^3: Prima + MCE::Hobo demonstration
by marioroy (Prior) on May 02, 2017 at 08:37 UTC

    For completeness, here's the code using Parallel::ForkManager. Calling MCE::Shared->init() is important to spread out IPC across 12 data channels. Not doing so, means all 200 workers share a single data channel and take much longer to run. Init is called automatically for MCE, MCE::Hobo and threads.

    Parallel::ForkManager

    use strict; use warnings; use Parallel::ForkManager; use MCE::Shared; use Time::HiRes 'time'; my $n = MCE::Shared->scalar(0); my $s = time; my $fork_manager = new Parallel::ForkManager(200); $fork_manager->set_waitpid_blocking_sleep(0); foreach my $child ( 1 .. 200 ) { my $pid = $fork_manager->start($child) and next; # init optionally takes an integer value MCE::Shared->init($child); my $v; $v = $n->incr() for 1 .. 933; $fork_manager->finish($child); } $fork_manager->wait_all_children; printf "duration: %0.3f\n", time - $s; print $n->get(), "\n"; __END__ main process, shared-manager process, and 200 workers Parallel::ForkManager: ~ 1200K per worker

    It's great knowing that MCE::Hobo isn't far behind regarding memory consumption. What I've learn is that simply loading modules doesn't tell the whole story. One must capture memory consumption while running for a better picture. :)

    Regards, Mario

      Hello mariojoy! I have one question I've been meaning to ask you about MCE::Shared, which setups up some IPC with filehandles passed thru a socketpair? Is this correct? I read in the docs about MCE using IO::FDPass. Can you elaborate on why you pass a filedescriptor, rather than just using the socketpair for communication? Can you elaborate on how this works, and any pitfalls it may have?

      I'm not really a human, but I play one on earth. ..... an animated JAPH

        Hi zentara,

        For MCE::Shared, the use of IO::FDPass applies to Condvar, Queue and to mce_open when constructing a shared handle from a non-shared handle not yet knowed to the shared-manager process. IO::FDpass isn't used for anything else.

        use MCE::Shared; # starting the shared-manager, implicitly my $ca = MCE::Shared->cache( max_keys => 500 ); # or starting early, explicitly MCE::Shared->start(); # at this point, IO::FDPass is necessary for constructing # a shared condvar or queue, passing fd descriptor # ... for new socket handle made during construction # workers block using a socket handle for ->wait, ->timedwait # the shared-manager unblocks by writing to _cw_sock my $cv = MCE::Shared->condvar(); # _cw_sock # ... for new socket handles made during construction # workers block using socket handles for ->dequeue, ->await # the shared-manager unblocks by writing to _qw_sock, _aw_sock my $q1 = MCE::Shared->queue(); # _qw_sock my $q2 = MCE::Shared->queue( await => 1 ); # _aw_sock

        Without IO::FDPass, one cannot construct a shared condvar or queue while the shared manager is running. The work around is to construct these before other shared objects (e.g. ->array, ->hash, etc). Then, start the shared-manager manually.

        use MCE::Shared; my $cv = MCE::Shared->condvar(0); my $que = MCE::Shared->queue( fast => 1 ); MCE::Shared->start() # must start manually my $hash = MCE::Shared->hash(); # or implicitly

        Note: MCE starts the shared-manager if not started. Ditto for MCE::Hobo because that constructs two shared hashes.

        Regarding mce_open, IO::FDPass is needed when constructing a shared-handle from a non-shared handle not yet available inside the shared-manager process. The workaround is having the non-shared handle made before the shared-manager process is started implicitly or explicitly.

        # the shared-manager knows of \*STDIN, \*STDOUT, \*STDERR mce_open my $shared_in, "<", \*STDIN; # ok mce_open my $shared_out, ">>", \*STDOUT; # ok mce_open my $shared_err, ">>", \*STDERR; # ok # but may not know of $non_shared_fh mce_open my $shared_fh, ">>", $non_shared_fh;

        Regards, Mario

Re^3: Prima + MCE::Hobo demonstration
by marioroy (Prior) on May 03, 2017 at 07:41 UTC

    MCE 1.829 and MCE::Shared 1.826 have been released. These include the updates for reducing memory consumption, especially for MCE::Shared::Server.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1189307]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (6)
As of 2024-03-29 05:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found