Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: running a function for different list parallely using fork

by hippo (Bishop)
on Mar 15, 2023 at 10:22 UTC ( [id://11150997]=note: print w/replies, xml ) Need Help??


in reply to running a function for different list parallely using fork

Have you considered Parallel::ForkManager? This is precisely the sort of problem it is designed to solve.


🦛

Replies are listed 'Best First'.
Re^2: running a function for different list parallely using fork
by ikegami (Patriarch) on Mar 15, 2023 at 16:22 UTC

    Exactly this.

    use Parallel::ForkManager qw( ); my @testLists = ...; my $pm = Parallel::ForkManager->new( 2 ); for my $list ( @testLists ) { $pm->start and next; $status += regressions( $opts, $list ); $pm->finish; } $pm->wait_all_children();

    That said, you are changing $status in the child to no effect. Did you mean to exit with that code?

    use Parallel::ForkManager qw( ); my @testLists = ...; my $status = 0; my $pm = Parallel::ForkManager->new( 2 ); $pm->run_on_finish(sub { my ( $pid, $exit_code, $ident ) = @_; $status += $exit_code; }); for my $list ( @testLists ) { $pm->start and next; my $status = regressions( $opts, $list ); $status = 255 if $status > 255; $pm->finish( $status ); } $pm->wait_all_children();

    P::FM also facilitates passing more complex values back to the parent.

    use Parallel::ForkManager qw( ); my @testLists = ...; my $status = 0; my $pm = Parallel::ForkManager->new( 2 ); $pm->run_on_finish(sub { my ( $pid, $exit_code, $ident, $exit_signal, $core_dump, $data ) = +@_; if ( $exit_code || $exit_signal || $core_dump ) { # Handle error. } $status += $data->{ status }; }); for my $list ( @testLists ) { $pm->start and next; my $status = regressions( $opts, $list ); $pm->finish( 0, { status => $status } ); } $pm->wait_all_children();

    Update: Applied marioroy's fix.

      See the Parallel::ForkManager demonstration, here. Something similar can be done with MCE::Child. Subsequently, MCE. Notice the PIDs in the 2nd demonstration where the workers persist.

      MCE::Child

      use strict; use warnings; use MCE::Child qw( ); use Time::HiRes qw( time ); my @testLists = qw( sun moon wind air ); my $opts = 'foo'; my $status = 0; sub regressions { my ($opts, $list) = @_; print "$$: $list\n"; sleep 1; # simulate processing return 1; } my $start = time(); MCE::Child->init( max_workers => 2, on_finish => sub { my ( $pid, $exit_code, $ident, $exit_signal, $core_dump, $data ) + = @_; if ( $exit_code || $exit_signal || $core_dump ) { # Handle error. } $status += $data->{ status }; } ); for my $list ( @testLists ) { MCE::Child->create(sub { my $status = regressions( $opts, $list ); return { status => 1 }; }); } MCE::Child->wait_all(); print "status: $status\n"; printf "duration: %0.3f seconds\n", time() - $start; __END__ 27481: sun 27482: moon 27483: wind 27484: air status: 4 duration: 2.013 seconds

      MCE

      use strict; use warnings; use MCE::Loop qw( ); use Time::HiRes qw( time ); my @testLists = qw( sun moon wind air ); my $opts = 'foo'; my $status = 0; sub regressions { my ($opts, $list) = @_; print "$$: $list\n"; sleep 1; # simulate processing return 1; } my $start = time(); MCE->new( max_workers => 2, chunk_size => 1, input_data => \@testLists, gather => sub { $status += $_[0] }, user_func => sub { my $list = $_; my $status = regressions( $opts, $list ); MCE->gather( $status ); }, )->run(); print "status: $status\n"; printf "duration: %0.3f seconds\n", time() - $start; __END__ 27562: sun 27563: moon 27562: wind 27563: air status: 4 duration: 2.005 seconds

        Indeed, I didn't mean to exclude other modules. I meant the wheel didn't need to be reinvented.

      The "$pm->run_on_finish" method takes 2 additional arguments; signal_code and core_dump. I tried the following variation, based on ikegami's solution.

      Parallel::ForkManager

      use strict; use warnings; use Parallel::ForkManager qw( ); use Time::HiRes qw( time ); my @testLists = qw( sun moon wind air ); my $opts = 'foo'; my $status = 0; sub regressions { my ($opts, $list) = @_; print "$$: $list\n"; sleep 1; # simulate processing return 1; } my $start = time(); my $pm = Parallel::ForkManager->new( 2 ); # P::FM by default, only waits for its own child processes, # which is what we want. # Refer to 'BLOCKING CALLS' in the module documentation. # Let's decrease the sleep period time (default 1.0). $pm->set_waitpid_blocking_sleep(0.25); $pm->run_on_finish(sub { my ( $pid, $exit_code, $ident, $exit_signal, $core_dump, $data ) = +@_; if ( $exit_code || $exit_signal || $core_dump ) { # Handle error. } $status += $data->{ status }; }); for my $list ( @testLists ) { $pm->start and next; my $status = regressions( $opts, $list ); $pm->finish( 0, { status => $status } ); } $pm->wait_all_children(); print "status: $status\n"; printf "duration: %0.3f seconds\n", time() - $start; __END__ 27470: sun 27471: moon 27472: wind 27473: air status: 4 duration: 2.257 seconds

        Thanks, fixed.

      One thing that is not clear to me, how it will handle the running of regressions in case number of lists in @teslLists is odd (i.e 3 or 5 or 7). because at last, only one list will be left. so will below line will run only last list even though we have passed 2 to run in parallel.

      my $pm = Parallel::ForkManager->new( 2 );

        P::FM forks a child for each "task". If you provide a limit (e.g. the 2), it waits for children to finish before forking if forking would bring the number of children over the limit.

        use v5.14; use warnings; use Parallel::ForkManager qw( ); use Time::HiRes qw( sleep ); my $pm = Parallel::ForkManager->new ( 2 ); for ( 1 .. 9 ) { $pm->start and next; say "[$_] start"; sleep( $_ == 1 ? 4 : 0.3 ); say "[$_] done"; $pm->finish; } $pm->wait_all_children();

        Annotated output:

        1 2 3 4 5 6 7 8 9 ----- --- --- --- --- --- --- --- --- --- [1] start 1 XXX [2] start 1 & 2 XXX XXX [2] done 1 XXX [3] start 1 & 3 XXX XXX [3] done 1 XXX [4] start 1 & 4 XXX XXX [4] done 1 XXX [5] start 1 & 5 XXX XXX [5] done 1 XXX [1] done [6] start 6 XXX [7] start 6 & 7 XXX XXX [6] done 7 XXX [7] done [8] start 8 XXX [9] start 8 & 9 XXX XXX [8] done 9 XXX [9] done

      Thank you, let me try!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11150997]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others studying the Monastery: (3)
As of 2024-03-29 01:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found