in reply to Graceful shutdowns and Parallel::ForkManager

I'm afraid that won't work for you.

Here's the implementation of P::FM::start (found by doing vi `perldoc -l Parallel::ForkManager`): 

sub start { my ($s,$identification)=@_;
  die "Cannot start another process while you are in the child process
+"
    if $s->{in_child};
  while ( ( keys %{ $s->{processes} } ) >= $s->{max_proc}) {
    $s->on_wait;
    $s->wait_one_child(defined $s->{on_wait_period} ? &WNOHANG : undef
+);
  };
  $s->wait_children;
  if ($s->{max_proc}) {
    my $pid=fork();
    die "Cannot fork: $!" if !defined $pid;
    if ($pid) {
      $s->{processes}->{$pid}=$identification;
      $s->on_start($pid,$identification);
    } else {
      $s->{in_child}=1 if !$pid;
    }
    return $pid;
  } else {
    $s->{processes}->{$$}=$identification;
    $s->on_start($$,$identification);
    return 0; # Simulating the child which returns 0
  }
}

[download]
If you look at the "} else {" branch, you'll see that if no processes are available (when $s->{max_proc}==0, what set_max_procs(0) does), the task is continued in the parent process.

That looks like a cool feature, actually, since you can test out your code in a single process loop before going parallel, if you like.

I think you should have your alarm sig set a flag, which you could check in your "my $data" loop: 

my $stopFlag=0;

$SIG{ARLM} = sub { $stopFlag=1; };

    alarm($MAX_TIME);
    foreach my $data (@all_data) {
        last if $stopFlag;

        my $pid = $pm->start and next;

        # Process or call processing routines 
        # for $data here

        $pm->finish;
    }
    $pm->wait_all_children;
    alarm(0);

[download]
Hmmm, I think that'd work... Not sure, though.
--
Mike

Replies are listed 'Best First'.
Re^2: Graceful shutdowns and Parallel::ForkManager
by atcroft (Abbot) on Jul 25, 2002 at 20:03 UTC

    Thanks for commenting, RMGir-I greatly appreciate the input.

    After reading your comment, I felt a little more confident that a test would not bork my box. The test code is below, but the results seemed to indicate that it actually did work, surprisingly enough. (I get a list of numbers, the "Who's left?" message, then what appears to be those in the queue at the time of the alarm.) I'm not sure which parts I'm reading incorrectly in the module's code that make it appear that it shouldn't work, though. Can someone verify that they get the same/similar results? 

    #!/usr/bin/perl -w

use Parallel::ForkManager;
use strict;
use vars qw($pm);

sub max_processes  { 5; }
sub max_seconds {
    ( ( ( ( ( 0 * 24 ) + 0 ) * 60 ) + 0 ) * 60 ) + 10;
}    # Time in seconds - 0d 0h 0m 10s

$pm = new Parallel::ForkManager(&max_processes);

$SIG{ALRM} = sub { die ("TimeOut"); };
eval {
    alarm( &max_seconds() );
    foreach my $data (0..500) {
        my $pid = $pm->start and next;

        print($data);
        sleep(int(rand(10)));

        print('-', $data, "\n");

        $pm->finish;
    }
        print("Waiting on everyone...\n");
    $pm->wait_all_children;
    alarm(0);
};
if ($@) {
    if ( $@ =~ m/TimeOut/ ) {
        # timed out; do what you will here
        $pm->set_max_procs(0);
        print("Who's left?\n");
        $pm->wait_all_children;
    }
    else {
        alarm(0);
        die;
    }
}

    [download]
    Results:
    2-2
3-3
6-6
5-5
8-8
4-4
9-9
7-7
12-12
0-0
10-10
1-1
11-11
Who's left?
14-14
15-15
16-16
13-13
17-17
[reply]
[d/l]
      just a quick comment...

      sub max_processes  { 5; }
      if you're trying to make this an inlined constant, you'll need to provide an empty prototype. i prefer making my constants ALL CAPS, and referring to them without a sigil. something like:

      sub MAX_PROCESSES() { 5 }

      and later...

      $pm = Parallel::ForkManager->new(MAX_PROCESSES);

      you'll find more info on this technique in perlsub.

      ~Particle *accelerates*

[reply]
[d/l]
[select]

        I appreciate the heads' up that I was doing that incorrectly, and the reference to where to find how to do so properly. I have only just started to use that technique, and now, thanks to your assistance, will know better how to use it properly.

[reply]
      D'oh. I completely missed that you were using the alarm to interrupt the eval.

      But since interrupting the eval kills your while loop, why bother with set_max_procs at all? No other children can be dispatched anyhow...
      --
      Mike

[reply]

        Because I want the children currently in-process to complete naturally, rather than being terminated before completing. (This also assumes that there will not be a failure which kills them, which could happen, but would be outside my control at that point.)

[reply]

In Section Seekers of Perl Wisdom