in reply to Re: Parallel::Forkmanager question
in thread Parallel::Forkmanager question

platform is Linux Red Hat. I am currently testing with 9 servers in the list. When I run the script without Parallel::Forkmanager what is happening is that the script will loop thru and process each server using the system call to scp a file. The script will wait for that system call to finish and then continue to the next server. It seems like that is exactly whats happening with the fork. It's processing 1 server at a time, unless I'm not seeing this correctly. The reason I had the wait for all children is that I want to make sure all the servers have been processed before continuing on. I am adding time stamps to the print statements to show what is happening. Here is the code and the output. 



my $concurrent_fork_limit = 9;
my $fork_manager = Parallel::ForkManager->new($concurrent_fork_limit);

foreach my $server (@servers) {

  my $pid = $fork_manager->start;

  if ($pid) {
    my ($dstamp, $hstamp) = &next_time();
    print "$dstamp $hstamp $$: child process started, with a key of $s
+erver ($pid)\n";
    print "\nWorking On Server ->  $server\n";
    print "Sending File      ->  $file\n";

    my $timeout = 20;
    eval {
       local $SIG{ALRM} = sub { die "alarm\n" };
       alarm $timeout;
       my $rc = system("scp -p $file $user\@$fqdn:$rpath/$fname");
       print "\nReturn Code For Server -> $server  File -> $file  = $r
+c\n\n\n";
       alarm 0;
    };
    if( $@ ) {
      $nogood{$server}++;
    }
  }
  $fork_manager->finish;
}

$fork_manager->wait_all_children();
my ($fstamp, $dstamp, $hstamp) = &next_time();
print "\n\n\n$dstamp $hstamp All Children Finished, Continuing\n\n";

[download]


04/06/2017 08:56:23 127266: child process started, with a key of server01 (127269)
Working On Server -> server01.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server01.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:25 127266: child process started, with a key of server02 (127273)
Working On Server -> server02.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server02.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:26 127266: child process started, with a key of server03 (127278)
Working On Server -> server03.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server03.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:28 127266: child process started, with a key of server04 (127282)
Working On Server -> server04.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server04.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:29 127266: child process started, with a key of server05 (127286)
Working On Server -> server05.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server05.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:31 127266: child process started, with a key of server06 (127290)
Working On Server -> server06.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server06.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:32 127266: child process started, with a key of server07 (127294)
Working On Server -> server07.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server07.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:34 127266: child process started, with a key of server08 (127298)
Working On Server -> server08.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server08.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:35 127266: child process started, with a key of server09 (127303)
Working On Server -> server09.domain.com
Sending File -> mkdir.pl
Return Code For Server -> server09.domain.com File -> mkdir.pl = 0


04/06/2017 08:56:35 All Children Finished, Continuing

Replies are listed 'Best First'.
Re^3: Parallel::Forkmanager question
by 1nickt (Canon) on Apr 06, 2017 at 13:49 UTC

    As your output shows, the same process is handling all the tasks. That's because the parent is forking a child but then continuing the work itself.

    You are missing the statement that causes the parent to skip out of the loop. See the doc for Parallel::ForkManager, which states (emphasis added):

    • Next, use $pm->start to do the fork. $pm returns 0 for the child process, and child pid for the parent process (see also "fork()" in perlfunc(1p)). The "and next" skips the internal loop in the parent process.

    The following example shows the difference: 

    use strict; use warnings; use feature 'say';
use Parallel::ForkManager;
use Time::HiRes qw/ time /;

$|++;

my $forker = Parallel::ForkManager->new(4);
say "v.1";
for ( 0 .. 9 ) {
    my $pid = $forker->start;
    if ( $pid ) {
        say "1 $$ $_ start: " . time;
        sleep 1;
    }
    $forker->finish;
}

my $forker2 = Parallel::ForkManager->new(4);
say "v.2";
for ( 0 .. 9 ) {
    my $pid = $forker->start and next;
    say "2 $$ $_ start: " . time;
    sleep 1;
    $forker->finish;
}

__END__

    [download]
    Output: 
    $ perl 1187274.pl

v.1
1 19015 0 start: 1491486225.44169
1 19015 1 start: 1491486226.44244
1 19015 2 start: 1491486227.44318
1 19015 3 start: 1491486228.44402
1 19015 4 start: 1491486229.44495
1 19015 5 start: 1491486230.44586
1 19015 6 start: 1491486231.447
1 19015 7 start: 1491486232.44791
1 19015 8 start: 1491486233.44877
1 19015 9 start: 1491486234.4497
v.2
2 19026 0 start: 1491486235.45079
2 19027 1 start: 1491486235.45113
2 19028 2 start: 1491486235.45138
2 19029 3 start: 1491486235.4517
2 19030 4 start: 1491486237.45457
2 19031 5 start: 1491486237.45492
2 19032 6 start: 1491486237.45516
2 19033 7 start: 1491486237.45543
2 19035 9 start: 1491486239.45953
2 19034 8 start: 1491486239.45992

    [download]

    Hope this helps!


    The way forward always starts with a minimal test.
[reply]
[d/l]
[select]

      I changed the code to include the 'start and next' and moved the finish within the if pid loop. However now I'm getting completely new error messages.

      Cannot start another process while you are in the child process at /usr/local/share/perl5/Parallel/ForkManager.pm line 580.
      Cannot start another process while you are in the child process at /usr/local/share/perl5/Parallel/ForkManager.pm line 580.
      child process '12087' disappeared. A call to `waitpid` outside of Parallel::ForkManager might have reaped it.
      child process '12088' disappeared. A call to `waitpid` outside of Parallel::ForkManager might have reaped it.

[reply]

      sorry, posted too soon. I had to take out the if pid and now its works great. Thanks for the help.

[reply]

In Section Seekers of Perl Wisdom