Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
Dear Perl monks,

may I ask you for your wisdom about the parallel processing in Mojolicios, please?

Here I am trying to understand how to implement parallel processes within a loop, set up arguments to the subprocess and get the result before the render (surely, I am using $c->render_later).

I was trying many different approaches and one seems to work for me, but the only thing with the argument (parameter). If I run the code below without ->($_) the processes work in parallel, the main sub waits for all of them to complete (which takes about 2 seconds) and then renders the page. But once I add ->($_) or any int, for example, the processes run in a blocking way - one by one and it takes about 12 seconds.

use Mojo::Base -strict, -signatures, -async_await; use Mojo::IOLoop::Subprocess; use Mojo::Promise; async sub testMethod{ $c->render_later; show "BEFORE"; my @promises; for(0..5){ say "(START) This is $_ loop"; my $subprocess = Mojo::IOLoop::Subprocess->new; push @promises, $subprocess->run_p( sub { my $i = shift || 1; my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Good bye, from $i and $$!"; $promise->resolve("Done for $i"); return $promise; #return "Done for $i"; }->($_) )->then(sub ($_result) { show $_result; return $_result; })->catch(sub { my $err = shift; say "Subprocess error: $err"; }); $subprocess->ioloop->start unless $subprocess->ioloop->is_runn +ing; say "(END) This is $_ loop"; } my @results = await Mojo::Promise->all_settled(@promises); show @results; show "AFTER"; return $c->render(text => "All done", status => 200); }


Below is the output with the parameter sent via ->($_)(works in a blocking way, takes about 12 secs):

======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 0 loop (START) This is 1 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 1 loop (START) This is 2 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 2 loop (START) This is 3 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 3 loop (START) This is 4 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 4 loop (START) This is 5 loop Hello, from 5 and 17430! Good bye, from 5 and 17430! (END) This is 5 loop Subprocess error: Can't locate object method "Promise=HASH(0xdc0f198)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc0ac00)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc06310)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdbff708)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc11650)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. Subprocess error: Can't locate object method "Promise=HASH(0xdc0fae0)" + via package "Mojo" at /home/ubuntu/perl5/perlbrew/perls/perl-5.26.3/ +lib/site_perl/5.26.3/Mojo/IOLoop/Subprocess.pm line 53. ======( @results )===========[ 'Test.pm', line 238 ]====== [ { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, { status => "fulfilled", value => [1] }, ] ======( "AFTER" )============[ 'Test.pm', line 247 ]====== "AFTER"


This is the output without parameters sent (no </code>->($_)</code> construction) (works parallel, takes about 2 secs):

======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop (END) This is 0 loop (START) This is 1 loop (END) This is 1 loop (START) This is 2 loop (END) This is 2 loop (START) This is 3 loop (END) This is 3 loop (START) This is 4 loop (END) This is 4 loop (START) This is 5 loop (END) This is 5 loop Hello, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037! Hello, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdbfd950) and 20033! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc043f8) and 20034! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09528) and 20035! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc09c00) and 20036! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0a2c0) and 20037! Good bye, from Mojo::IOLoop::Subprocess=HASH(0xdc0e570) and 20038! ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( $_result )===========[ 'Test.pm', line 213 ]====== undef ======( @results )===========[ 'Test.pm', line 225 ]====== [ { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, { status => "fulfilled", value => [undef] }, ] ======( "AFTER" )============[ 'Test.pm', line 234 ]====== "AFTER"


Another option I have tried:

sub { my $sp = shift; # $subprocess my $i = shift || 3; #my $promise = Mojo::Promise->new; say "Hello, from $i and $$!"; sleep 2; say "Goodbye, from $i and $$!"; #$promise->resolve("Done for $i"); #return $promise; return "Done for $i"; #}->($_) }


This gives the next output (works parallel, takes about 2 secs):

======( "BEFORE" )===========[ 'Test.pm', line 189 ]====== "BEFORE" (START) This is 0 loop (END) This is 0 loop (START) This is 1 loop (END) This is 1 loop (START) This is 2 loop (END) This is 2 loop (START) This is 3 loop (END) This is 3 loop (START) This is 4 loop (END) This is 4 loop (START) This is 5 loop (END) This is 5 loop Hello, from 3 and 20608! Hello, from 3 and 20609! Hello, from 3 and 20610! Hello, from 3 and 20611! Hello, from 3 and 20612! Hello, from 3 and 20613! Good bye, from 3 and 20608! Good bye, from 3 and 20611! Good bye, from 3 and 20609! Good bye, from 3 and 20610! Good bye, from 3 and 20612! Good bye, from 3 and 20613! ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( $_result )===========[ 'Test.pm', line 216 ]====== "Done for 3" ======( @results )===========[ 'Test.pm', line 228 ]====== [ { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, { status => "fulfilled", value => ["Done for 3"] }, ] ======( "AFTER" )============[ 'Test.pm', line 237 ]====== "AFTER"
Dear monks, I spent 3 days on my own trying to get this working, but cannot. I need to run parallel processes with a custom parameter for each process, all processes run from the loop (or a recursive sub), wait until they all are done, and then have a result from each completed process and finally render the page.

Please, help me with this.

I want to get an idea of what I do wrong and how to make it work, please!

Thank you in advance!
Peace!

In reply to use Mojo::IOLoop::Subprocess - cannot pass argument to the subprocess by PerlDiver

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others goofing around in the Monastery: (9)
As of 2024-03-28 11:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found