Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: Code brewing for the upcoming MCE 10 year anniversary

by marioroy (Prior)
on Oct 23, 2022 at 13:45 UTC ( [id://11147608]=note: print w/replies, xml ) Need Help??


in reply to Code brewing for the upcoming MCE 10 year anniversary

Behind the scene, I'm using MCE (chunk_size set to 1).
mce_foreach my $var ( @list ) { ... } mce_foreach ( @list ) { ... } mce_foreach_f my $var ( $path or $fh ) { ... } mce_foreach_f ( $path or $fh ) { ... } mce_foreach_s my $var ( @list ) { ... } mce_foreach_s ( @list ) { ... }

The best part is the spawn and sync keywords.

# context scalar, array, or hash spawn (my|our)* $var @var or %var = sub { codeblock }; spawn (my|our)* $var @var or %var = sub { func(@_) }, args; spawn (my|our)* $var @var or %var = func(arg1, ...); spawn (my|our)* $var @var or %var = func(); sync $var; sync @var; sync %var; # anonymous code blocks spawn $identVar or "quotedString", sub { codeblock }; spawn sub { codeblock }; sync;

Replies are listed 'Best First'.
Re^2: Code brewing for the upcoming MCE 10 year anniversary
by marioroy (Prior) on Oct 23, 2022 at 14:25 UTC

    Sometimes, the use-case calls for processing at the chunk level inside the code block. For example, the worker receives the chunk as a whole.

    mce_forchunk mce_forchunk_f mce_forchunk_s

    I will use MCE::Loop behind the scene.

      The mce_forchunk keywords are working. Sometimes, it's preferably to process at the chunk level. So, here it is.

      mce_forchunk

      use MCE::Simple -strict, max_workers => 4, init_relay => ''; mce_forchunk (11..19) { my ($mce, $chunk_ref, $chunk_id) = @_; for my $i (@{ $chunk_ref }) { MCE->say($i); } } mce_forchunk (31..39) { for my $i (@{ $_ }) { MCE->say($i); } } mce_forchunk my $chunk (51..59) { for my $i (@{ $chunk }) { MCE->say($i); } } mce_forchunk my $chunk (71..79) { my $output = "From chunk_id: ".MCE->chunk_id."\n"; for my $i (@{ $chunk }) { $output .= "$i\n"; } MCE::relay { # write directly to STDOUT, not involve manager process print $output; }; }

      Output

      15 13 11 16 14 17 12 18 19 33 34 31 39 32 37 35 38 36 51 57 53 52 55 58 54 56 59 From chunk_id: 1 71 72 From chunk_id: 2 73 74 From chunk_id: 3 75 76 From chunk_id: 4 77 78 From chunk_id: 5 79

      Spawn MCE asynchronously

      MCE::Simple expands keywords at compile time to parallel code.

      use MCE::Simple ( include => [qw/ -strict -signatures /], max_workers => 4, init_relay => '', ); MCE::Simple->init( on_finish => sub ( $pid, $exit, $ident, $signal, $error, @ret ) { say "MCE job ** $ident ** completed; status $exit."; print join('', @ret); }, ); spawn "foo", sub { my @results; MCE::Simple->init(gather => \@results); mce_forchunk my $chunk (91..99) { my $output = "From chunk_id: ".MCE->chunk_id."\n"; for my $i (@{ $chunk }) { $output .= "$i\n"; } MCE::relay { MCE->gather($output); }; } @results; }; # do something else sync;

      Output

      MCE job ** foo ** completed; status 0. From chunk_id: 1 91 92 From chunk_id: 2 93 94 From chunk_id: 3 95 96 From chunk_id: 4 97 98 From chunk_id: 5 99

        Or using MCE::Loop instead. One can have MCE::Loop gather orderly via MCE::relay.

        use MCE::Simple ( include => [qw/ -strict -signatures -mce_loop /], max_workers => 4, init_relay => '', ); MCE::Simple->init( on_finish => sub ( $pid, $exit, $ident, $signal, $error, @ret ) { say "MCE job ** $ident ** completed; status $exit."; print join('', @ret); }, ); spawn "foo", sub { my @results = mce_loop { my $output = "From chunk_id: ".MCE->chunk_id."\n"; for my $i (@{ $_ }) { $output .= "$i\n"; } MCE::relay { MCE->gather($output); }; } 91..99; }; # do something else sync;

        Thank you, PerlMonks. I was able to think out loud. The demonstrations work flawlessly.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11147608]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2024-04-24 02:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found