in reply to running a function for different list parallely using fork

#!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11150996
use warnings;

my @testLists = 1 .. 7; # FAKE for testing
my $opts = 'opts'; # FAKE for testing
my %pids; # NOTE changed to hash because children finish in any order

foreach my $list (@testLists) 
  {
  while( keys %pids >= 2 )
    {
    delete $pids{ +wait };
    }
  if( my $pid = fork )
    {
    $pids{ $pid }++;
    }
  elsif( defined $pid )
    {
    my $status += regressions($opts, $list);
    exit;
    }
  else
    {
    die "fork failed $!";
    }
  }
foreach my $pid ( keys %pids ) 
  {
  waitpid($pid, 0);
  }

sub regressions # FIXME for testing
  {
  print time, " pid $$ starting on (@_)\n";
  sleep 1 + int rand 3;
  print time, " pid $$ ending on (@_)\n";
  }

[download]

Sample Output:

1678877305 pid 65651 starting on (opts 1)
1678877305 pid 65652 starting on (opts 2)
1678877307 pid 65651 ending on (opts 1)
1678877307 pid 65653 starting on (opts 3)
1678877308 pid 65652 ending on (opts 2)
1678877308 pid 65653 ending on (opts 3)
1678877308 pid 65654 starting on (opts 4)
1678877308 pid 65655 starting on (opts 5)
1678877310 pid 65655 ending on (opts 5)
1678877310 pid 65656 starting on (opts 6)
1678877311 pid 65654 ending on (opts 4)
1678877311 pid 65657 starting on (opts 7)
1678877312 pid 65656 ending on (opts 6)
1678877313 pid 65657 ending on (opts 7)

[download]

Replies are listed 'Best First'.
Re^2: running a function for different list parallely using fork - MCE::Channel
by marioroy (Prior) on Mar 16, 2023 at 04:13 UTC

    Tybalt89's solution is neat. Here is a variation using MCE::Channel where the child sends the status to the manager process. The manager does a non-blocking read in the event the child crashed or exited early; e.g. not send the status.

    MCE::Channel

    #!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11150996
use warnings;
use MCE::Channel;

my @testLists = 1 .. 7; # FAKE for testing
my $opts = 'opts'; # FAKE for testing
my %pids; # NOTE changed to hash because children finish in any order
my $status = 0;

my $chnl = MCE::Channel->new();

foreach my $list (@testLists) 
  {
  while( keys %pids >= 2 )
    {
    delete $pids{ +wait };
    my $val = $chnl->recv_nb();
    $status += $val if defined($val);
    }
  if( my $pid = fork )
    {
    $pids{ $pid }++;
    }
  elsif( defined $pid )
    {
    my $status = regressions($opts, $list);
    $chnl->send($status);
    exit;
    }
  else
    {
    die "fork failed $!";
    }
  }
foreach my $pid ( keys %pids ) 
  {
  waitpid($pid, 0);
  my $val = $chnl->recv_nb();
  $status += $val if defined($val);
  }

print "status: $status\n";

sub regressions # FIXME for testing
  {
  print time, " pid $$ starting on (@_)\n";
  sleep 1 + int rand 3;
  print time, " pid $$ ending on (@_)\n";
  return 1;
  }

__END__
1678939864 pid 27850 starting on (opts 1)
1678939864 pid 27851 starting on (opts 2)
1678939866 pid 27851 ending on (opts 2)
1678939866 pid 27852 starting on (opts 3)
1678939867 pid 27850 ending on (opts 1)
1678939867 pid 27853 starting on (opts 4)
1678939868 pid 27853 ending on (opts 4)
1678939868 pid 27854 starting on (opts 5)
1678939869 pid 27852 ending on (opts 3)
1678939869 pid 27855 starting on (opts 6)
1678939871 pid 27855 ending on (opts 6)
1678939871 pid 27854 ending on (opts 5)
1678939871 pid 27856 starting on (opts 7)
1678939872 pid 27856 ending on (opts 7)
status: 7

    [download]
[reply]
[d/l]
Re^2: running a function for different list parallely using fork
by tybalt89 (Monsignor) on Mar 17, 2023 at 01:25 UTC

    And here's the Forking::Amazing version. See Re: figuring out Parallel::ForkManager slots for the Forking::Amazing module, which I wrote because I didn't particularly care for Parallel::ForkManager. I think Forking::Amazing makes it clearer what runs in the child and what runs in the parent, and makes it easier to pass results from the child back to the parent. 

    #!/usr/bin/perl

use strict; # https://perlmonks.org/?node_id=11150996
use warnings;
use Forking::Amazing;

my @testLists = 1 .. 10; # FAKE for testing
my $opts = 'opts'; # FAKE for testing
my $status = 0;

Forking::Amazing::run
  2,                                          # max forks
  sub { return [ regressions($opts, @_) ]; }, # child
  sub { $status += $_[1][0]; },               # parent
  @testLists;                                 # argument for each fork
print "status $status\n";

sub regressions # FIXME for testing
  {
  print time, " pid $$ starting on (@_)\n";
  select undef, undef, undef, 1 + rand 3;
  print time, " pid $$ ending on (@_)\n";
  return pop;
  }

    [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom