Dear Perl monks,
may I ask you for your wisdom about the parallel processing in Mojolicios, please?
Here I am trying to understand how to implement parallel processes within a loop, set up arguments to the subprocess and get the result before the render (surely, I am using
$c->render_later).
I was trying many different approaches and one seems to work for me, but the only thing with the argument (parameter). If I run the code below without
->($_) the processes work in parallel, the main sub waits for all of them to complete (which takes about 2 seconds) and then renders the page. But once I add
->($_) or any int, for example, the processes run in a blocking way - one by one and it takes about 12 seconds.
use Mojo::Base -strict, -signatures, -async_await;
use Mojo::IOLoop::Subprocess;
use Mojo::Promise;
async sub testMethod{
$c->render_later;
show "BEFORE";
my @promises;
for(0..5){
say "(START) This is $_ loop";
my $subprocess = Mojo::IOLoop::Subprocess->new;
push @promises, $subprocess->run_p(
sub {
my $i = shift || 1;
my $promise = Mojo::Promise->new;
say "Hello, from $i and $$!";
sleep 2;
say "Good bye, from $i and $$!";
$promise->resolve("Done for $i");
return $promise;
#return "Done for $i";
}->($_)
)->then(sub ($_result) {
show $_result;
return $_result;
})->catch(sub {
my $err = shift;
say "Subprocess error: $err";
});
$subprocess->ioloop->start unless $subprocess->ioloop->is_runn
+ing;
say "(END) This is $_ loop";
}
my @results = await Mojo::Promise->all_settled(@promises);
show @results;
show "AFTER";
return $c->render(text => "All done", status => 200);
}
Below is the output
with the parameter sent via
->($_)(works in a blocking way, takes about 12 secs):
======( "BEFORE" )===========[ 'Test.pm', line 189 ]======
"BEFORE"
(START) This is 0 loop
Hello, from 5 and 17430!
Good bye, from 5 and 17430!
(END) This is 0 loop
(START) This is 1 loop
Hello, from 5 and 17430!
Good bye, from 5 and 17430!
(END) This is 1 loop
(START) This is 2 loop
Hello, from 5 and 17430!
Good bye, from 5 and 17430!
(END) This is 2 loop
(START) This is 3 loop
Hello, from 5 and 17430!
Good bye, from 5 and 17430!
(END) This is 3 loop
(START) This is 4 loop
Hello, from 5 and 17430!
Good bye, from 5 and 17430!
(END) This is 4 loop
(START) This is 5 loop
Hello, from 5 and 17430!
Good bye, from 5 and 17430!
(END) This is 5 loop
Subprocess error: Can't locate object method "Promise=HASH(0xdc0f198)"
+ via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/
+lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53.
Subprocess error: Can't locate object method "Promise=HASH(0xdc0ac00)"
+ via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/
+lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53.
Subprocess error: Can't locate object method "Promise=HASH(0xdc06310)"
+ via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/
+lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53.
Subprocess error: Can't locate object method "Promise=HASH(0xdbff708)"
+ via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/
+lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53.
Subprocess error: Can't locate object method "Promise=HASH(0xdc11650)"
+ via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/
+lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53.
Subprocess error: Can't locate object method "Promise=HASH(0xdc0fae0)"
+ via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/
+lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53.
======( @results )===========[ 'Test.pm', line 238 ]======
[
{ status => "fulfilled", value => [1] },
{ status => "fulfilled", value => [1] },
{ status => "fulfilled", value => [1] },
{ status => "fulfilled", value => [1] },
{ status => "fulfilled", value => [1] },
{ status => "fulfilled", value => [1] },
]
======( "AFTER" )============[ 'Test.pm', line 247 ]======
"AFTER"
This is the output
without parameters sent (no </code>->($_)</code> construction) (works parallel, takes about 2 secs):
======( "BEFORE" )===========[ 'Test.pm', line 189 ]======
"BEFORE"
(START) This is 0 loop
(END) This is 0 loop
(START) This is 1 loop
(END) This is 1 loop
(START) This is 2 loop
(END) This is 2 loop
(START) This is 3 loop
(END) This is 3 loop
(START) This is 4 loop
(END) This is 4 loop
(START) This is 5 loop
(END) This is 5 loop
Hello, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033!
Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034!
Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036!
Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035!
Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037!
Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038!
Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033!
Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034!
Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035!
Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036!
Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037!
Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038!
======( $_result )===========[ 'Test.pm', line 213 ]======
undef
======( $_result )===========[ 'Test.pm', line 213 ]======
undef
======( $_result )===========[ 'Test.pm', line 213 ]======
undef
======( $_result )===========[ 'Test.pm', line 213 ]======
undef
======( $_result )===========[ 'Test.pm', line 213 ]======
undef
======( $_result )===========[ 'Test.pm', line 213 ]======
undef
======( @results )===========[ 'Test.pm', line 225 ]======
[
{ status => "fulfilled", value => [undef] },
{ status => "fulfilled", value => [undef] },
{ status => "fulfilled", value => [undef] },
{ status => "fulfilled", value => [undef] },
{ status => "fulfilled", value => [undef] },
{ status => "fulfilled", value => [undef] },
]
======( "AFTER" )============[ 'Test.pm', line 234 ]======
"AFTER"
Another option I have tried:
sub {
my $sp = shift; # $subprocess
my $i = shift || 3;
#my $promise = Mojo::Promise->new;
say "Hello, from $i and $$!";
sleep 2;
say "Goodbye, from $i and $$!";
#$promise->resolve("Done for $i");
#return $promise;
return "Done for $i";
#}->($_)
}
This gives the next output (works parallel, takes about 2 secs):
======( "BEFORE" )===========[ 'Test.pm', line 189 ]======
"BEFORE"
(START) This is 0 loop
(END) This is 0 loop
(START) This is 1 loop
(END) This is 1 loop
(START) This is 2 loop
(END) This is 2 loop
(START) This is 3 loop
(END) This is 3 loop
(START) This is 4 loop
(END) This is 4 loop
(START) This is 5 loop
(END) This is 5 loop
Hello, from 3 and 20608!
Hello, from 3 and 20609!
Hello, from 3 and 20610!
Hello, from 3 and 20611!
Hello, from 3 and 20612!
Hello, from 3 and 20613!
Good bye, from 3 and 20608!
Good bye, from 3 and 20611!
Good bye, from 3 and 20609!
Good bye, from 3 and 20610!
Good bye, from 3 and 20612!
Good bye, from 3 and 20613!
======( $_result )===========[ 'Test.pm', line 216 ]======
"Done for 3"
======( $_result )===========[ 'Test.pm', line 216 ]======
"Done for 3"
======( $_result )===========[ 'Test.pm', line 216 ]======
"Done for 3"
======( $_result )===========[ 'Test.pm', line 216 ]======
"Done for 3"
======( $_result )===========[ 'Test.pm', line 216 ]======
"Done for 3"
======( $_result )===========[ 'Test.pm', line 216 ]======
"Done for 3"
======( @results )===========[ 'Test.pm', line 228 ]======
[
{ status => "fulfilled", value => ["Done for 3"] },
{ status => "fulfilled", value => ["Done for 3"] },
{ status => "fulfilled", value => ["Done for 3"] },
{ status => "fulfilled", value => ["Done for 3"] },
{ status => "fulfilled", value => ["Done for 3"] },
{ status => "fulfilled", value => ["Done for 3"] },
]
======( "AFTER" )============[ 'Test.pm', line 237 ]======
"AFTER"
Dear monks, I spent 3 days on my own trying to get this working, but cannot.
I need to run parallel processes with a custom parameter for each process, all processes run from the loop (or a recursive sub), wait until they all are done, and then have a result from each completed process and finally render the page.
Please, help me with this.
I want to get an idea of what I do wrong and how to make it work, please!
Thank you in advance!
Peace!
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.