http://qs1969.pair.com?node_id=11151265

PerlDiver has asked for the wisdom of the Perl Monks concerning the following question:

Dear Perl monks,

may I ask you for your wisdom about the parallel processing in Mojolicios, please?

Here I am trying to understand how to implement parallel processes within a loop, set up arguments to the subprocess and get the result before the render (surely, I am using $c->render_later).

I was trying many different approaches and one seems to work for me, but the only thing with the argument (parameter). If I run the code below without ->($_) the processes work in parallel, the main sub waits for all of them to complete (which takes about 2 seconds) and then renders the page. But once I add ->($_) or any int, for example, the processes run in a blocking way - one by one and it takes about 12 seconds.

use Mojo::Base -strict, -signatures, -async_await; use Mojo::IOLoop::Subprocess; use Mojo::Promise; async sub testMethod{ $c->render_later; show "BEFORE"; my @promises; for(0..5){ say "(START) This is $_ loop"; my $subprocess = Mojo::IOLoop::Subprocess->new; push @promises, $subprocess->run_p( sub { my $i = shift || 1; my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Good bye, from $i and $$!"; $promise->resolve("Done for $i"); return $promise; #return "Done for $i"; }->($_) )->then(sub ($_result) { show $_result; return $_result; })->catch(sub { my $err = shift; say "Subprocess error: $err"; }); $subprocess->ioloop->start unless $subprocess->ioloop->is_runn +ing; say "(END) This is $_ loop"; } my @results = await Mojo::Promise->all_settled(@promises); show @results; show "AFTER"; return $c->render(text => "All done", status => 200); }


Below is the output with the parameter sent via ->($_)(works in a blocking way, takes about 12 secs):

======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 0 loop (START) This is 1 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 1 loop (START) This is 2 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 2 loop (START) This is 3 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 3 loop (START) This is 4 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 4 loop (START) This is 5 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 5 loop Subprocess error: Can't locate object method "Promise=HASH(0xdc0f198)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc0ac00)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc06310)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdbff708)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc11650)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc0fae0)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. ======( @results )===========[ 'Test.pm', line 238 ]====== [ { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, ] ======( "AFTER" )============[ 'Test.pm', line 247 ]====== "AFTER"


This is the output without parameters sent (no </code>->($_)</code> construction) (works parallel, takes about 2 secs):

======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop (END) This is 0 loop (START) This is 1 loop (END) This is 1 loop (START) This is 2 loop (END) This is 2 loop (START) This is 3 loop (END) This is 3 loop (START) This is 4 loop (END) This is 4 loop (START) This is 5 loop (END) This is 5 loop Hello, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038! ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( @results )===========[ 'Test.pm', line 225 ]====== [ { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, ] ======( "AFTER" )============[ 'Test.pm', line 234 ]====== "AFTER"


Another option I have tried:

sub { my $sp = shift; # $subprocess my $i = shift || 3; #my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Goodbye, from $i and $$!"; #$promise->resolve("Done for $i"); #return $promise; return "Done for $i"; #}->($_) }


This gives the next output (works parallel, takes about 2 secs):

======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop (END) This is 0 loop (START) This is 1 loop (END) This is 1 loop (START) This is 2 loop (END) This is 2 loop (START) This is 3 loop (END) This is 3 loop (START) This is 4 loop (END) This is 4 loop (START) This is 5 loop (END) This is 5 loop Hello, from 3 and 20608! Hello, from 3 and 20609! Hello, from 3 and 20610! Hello, from 3 and 20611! Hello, from 3 and 20612! Hello, from 3 and 20613! Good bye, from 3 and 20608! Good bye, from 3 and 20611! Good bye, from 3 and 20609! Good bye, from 3 and 20610! Good bye, from 3 and 20612! Good bye, from 3 and 20613! ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( @results )===========[ 'Test.pm', line 228 ]====== [ { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, ] ======( "AFTER" )============[ 'Test.pm', line 237 ]====== "AFTER"
Dear monks, I spent 3 days on my own trying to get this working, but cannot. I need to run parallel processes with a custom parameter for each process, all processes run from the loop (or a recursive sub), wait until they all are done, and then have a result from each completed process and finally render the page.

Please, help me with this.

I want to get an idea of what I do wrong and how to make it work, please!

Thank you in advance!
Peace!

Replies are listed 'Best First'.
Re: use Mojo::IOLoop::Subprocess - cannot pass argument to the subprocess
by haukex (Archbishop) on Mar 29, 2023 at 10:15 UTC
    I want to get an idea of what I do wrong

    The problem with your first piece of code boils down to sub {}->(). This creates an anonymous sub and then immediately calls it, and ->run_p() gets the return value of this sub as its argument instead of the sub itself, as it should.

    This shows how important SSCCE along with the Basic debugging checklist are, especially boiling down the code and adding more print statements. If you inspect the output of the following code, you'll see how the subs are being called immediately in the same process as the IO loop.

    use Mojo::Base -strict, -signatures; use Mojo::IOLoop; Mojo::IOLoop->next_tick(sub ($loop) { say time-$^T, "s Loop PID is $$"; for my $i (0..1) { say time-$^T, "s Start of loop $i"; $loop->subprocess->run_p( sub ($j) { say time-$^T, "s Hello, from loop $i (arg $j) in PID $$!"; sleep 2; say time-$^T, "s Good bye, from PID $$!"; return "Return value of $i"; }->($i) )->then(sub ($result) { say time-$^T, "s Result from loop $i: $result"; })->catch(sub ($err) { say time-$^T, "s Subprocess error: $err"; }); say time-$^T, "s End of loop $i"; } }); Mojo::IOLoop->start;

    Output:

    0s Loop PID is 8353 0s Start of loop 0 0s Hello, from loop 0 (arg 0) in PID 8353! 2s Good bye, from PID 8353! 2s End of loop 0 2s Start of loop 1 2s Hello, from loop 1 (arg 1) in PID 8353! 4s Good bye, from PID 8353! 4s End of loop 1 4s Subprocess error: Can't locate object method "Return value of 0" vi +a package "Mojo::IOLoop::Subprocess" at /opt/perl5.28/lib/site_perl/5 +.28.1/Mojo/IOLoop/Subprocess.pm line 53. 4s Subprocess error: Can't locate object method "Return value of 1" vi +a package "Mojo::IOLoop::Subprocess" at /opt/perl5.28/lib/site_perl/5 +.28.1/Mojo/IOLoop/Subprocess.pm line 53.
    and how to make it work, please!

    Some frameworks allow passing arguments to functions like this, but I don't see anything in the Mojo::IOLoop::Subprocess documentation to indicate that Mojo does. So the answer is as you have already discovered: closures. If you have a complex function that you want to call with different arguments in the subprocess, then simply write it as a separate sub and call it from the closure (e.g. ->run_p(sub { my_complex_func($arg1, $arg2) })).

    I already laid out the above code as a closure, so all you need to do for it to work is simply remove the ->($i) in the above, and the output is what you probably expect:

    0s Loop PID is 8462 0s Start of loop 0 0s End of loop 0 0s Start of loop 1 0s End of loop 1 0s Hello, from loop 0 (arg Mojo::IOLoop::Subprocess=HASH(0x55ba4daf953 +0)) in PID 8463! 0s Hello, from loop 1 (arg Mojo::IOLoop::Subprocess=HASH(0x55ba4eebd13 +8)) in PID 8464! 2s Good bye, from PID 8463! 2s Good bye, from PID 8464! 2s Result from loop 0: Return value of 0 2s Result from loop 1: Return value of 1
      Thank you very much for your help! indeed, it works now!
Re: use Mojo::IOLoop::Subprocess - cannot pass argument to the subprocess
by PerlDiver (Initiate) on Mar 27, 2023 at 17:50 UTC
    I have done it with closure. Sure, I tried it at the first and it did not work, so I started searching for an alternative (this is how I came to the approach in the OP).

    The problem is with the for() loops. I cannot use in the subprocess any var, crated within the loop for() even $_).

    I am not sure I understand - why :)

    What was the remedy:

    my @looper = (0..5); foreach my $lp (@looper){ ... subprocess preparation as in the OP ... sub { my $i = $lp || 5; return $i; } }


    Now it shows every number from the @looper.

    The complete working example:

    use Mojo::Base -strict, -signatures, -async_await; use Mojo::IOLoop::Subprocess; use Mojo::Promise; async sub testMehod{ $c->render_later; my $start_timer1 = Time::HiRes::gettimeofday(); show "BEFORE"; my @promises; my @looper = (0..5); foreach my $lp (@looper){ say "(START) This is $lp loop"; my $subprocess = Mojo::IOLoop::Subprocess->new; push @promises, $subprocess->run_p( sub { my $i = $lp || 5; my $sp = shift; # $subprocess #my $i = shift || 3; #my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Good bye, from $i and $$!"; #$promise->resolve("Done for $i"); #return $promise; return "Done for $i"; #}->($_) } )->then(sub ($_result) { show $_result; return $_result; })->catch(sub { my $err = shift; say "Subprocess error: $err"; }); $subprocess->ioloop->start unless $subprocess->ioloop->is_runn +ing; say "(END) This is $lp loop"; } my @results = await Mojo::Promise->all_settled(@promises); show @results; my $stop_timer1 = Time::HiRes::gettimeofday(); my $total1 = sprintf("%.5f\n", $stop_timer1 - $start_timer1); return $c->render(text => "All done. Parallel: within $total1", st +atus => 405); }


    If you know - why it did not work with for() loop - please, kindly, share this knowledge :)

    Thank you!