PerlDiver has asked for the wisdom of the Perl Monks concerning the following question:
Dear Perl monks,
may I ask you for your wisdom about the parallel processing in Mojolicios, please?
Here I am trying to understand how to implement parallel processes within a loop, set up arguments to the subprocess and get the result before the render (surely, I am using $c->render_later).
I was trying many different approaches and one seems to work for me, but the only thing with the argument (parameter). If I run the code below without ->($_) the processes work in parallel, the main sub waits for all of them to complete (which takes about 2 seconds) and then renders the page. But once I add ->($_) or any int, for example, the processes run in a blocking way - one by one and it takes about 12 seconds.
Below is the output with the parameter sent via ->($_)(works in a blocking way, takes about 12 secs):
This is the output without parameters sent (no </code>->($_)</code> construction) (works parallel, takes about 2 secs):
Another option I have tried:
This gives the next output (works parallel, takes about 2 secs):
Please, help me with this.
I want to get an idea of what I do wrong and how to make it work, please!
Thank you in advance!
Peace!
may I ask you for your wisdom about the parallel processing in Mojolicios, please?
Here I am trying to understand how to implement parallel processes within a loop, set up arguments to the subprocess and get the result before the render (surely, I am using $c->render_later).
I was trying many different approaches and one seems to work for me, but the only thing with the argument (parameter). If I run the code below without ->($_) the processes work in parallel, the main sub waits for all of them to complete (which takes about 2 seconds) and then renders the page. But once I add ->($_) or any int, for example, the processes run in a blocking way - one by one and it takes about 12 seconds.
use Mojo::Base -strict, -signatures, -async_await; use Mojo::IOLoop::Subprocess; use Mojo::Promise; async sub testMethod{ $c->render_later; show "BEFORE"; my @promises; for(0..5){ say "(START) This is $_ loop"; my $subprocess = Mojo::IOLoop::Subprocess->new; push @promises, $subprocess->run_p( sub { my $i = shift || 1; my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Good bye, from $i and $$!"; $promise->resolve("Done for $i"); return $promise; #return "Done for $i"; }->($_) )->then(sub ($_result) { show $_result; return $_result; })->catch(sub { my $err = shift; say "Subprocess error: $err"; }); $subprocess->ioloop->start unless $subprocess->ioloop->is_runn +ing; say "(END) This is $_ loop"; } my @results = await Mojo::Promise->all_settled(@promises); show @results; show "AFTER"; return $c->render(text => "All done", status => 200); }
Below is the output with the parameter sent via ->($_)(works in a blocking way, takes about 12 secs):
======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 0 loop (START) This is 1 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 1 loop (START) This is 2 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 2 loop (START) This is 3 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 3 loop (START) This is 4 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 4 loop (START) This is 5 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 5 loop Subprocess error: Can't locate object method "Promise=HASH(0xdc0f198)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc0ac00)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc06310)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdbff708)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc11650)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc0fae0)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. ======( @results )===========[ 'Test.pm', line 238 ]====== [ { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, ] ======( "AFTER" )============[ 'Test.pm', line 247 ]====== "AFTER"
This is the output without parameters sent (no </code>->($_)</code> construction) (works parallel, takes about 2 secs):
======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop (END) This is 0 loop (START) This is 1 loop (END) This is 1 loop (START) This is 2 loop (END) This is 2 loop (START) This is 3 loop (END) This is 3 loop (START) This is 4 loop (END) This is 4 loop (START) This is 5 loop (END) This is 5 loop Hello, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038! ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( @results )===========[ 'Test.pm', line 225 ]====== [ { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, ] ======( "AFTER" )============[ 'Test.pm', line 234 ]====== "AFTER"
Another option I have tried:
sub { my $sp = shift; # $subprocess my $i = shift || 3; #my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Goodbye, from $i and $$!"; #$promise->resolve("Done for $i"); #return $promise; return "Done for $i"; #}->($_) }
This gives the next output (works parallel, takes about 2 secs):
Dear monks, I spent 3 days on my own trying to get this working, but cannot. I need to run parallel processes with a custom parameter for each process, all processes run from the loop (or a recursive sub), wait until they all are done, and then have a result from each completed process and finally render the page.======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop (END) This is 0 loop (START) This is 1 loop (END) This is 1 loop (START) This is 2 loop (END) This is 2 loop (START) This is 3 loop (END) This is 3 loop (START) This is 4 loop (END) This is 4 loop (START) This is 5 loop (END) This is 5 loop Hello, from 3 and 20608! Hello, from 3 and 20609! Hello, from 3 and 20610! Hello, from 3 and 20611! Hello, from 3 and 20612! Hello, from 3 and 20613! Good bye, from 3 and 20608! Good bye, from 3 and 20611! Good bye, from 3 and 20609! Good bye, from 3 and 20610! Good bye, from 3 and 20612! Good bye, from 3 and 20613! ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( @results )===========[ 'Test.pm', line 228 ]====== [ { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, ] ======( "AFTER" )============[ 'Test.pm', line 237 ]====== "AFTER"
Please, help me with this.
I want to get an idea of what I do wrong and how to make it work, please!
Thank you in advance!
Peace!
Back to
Seekers of Perl Wisdom