atcroft has asked for the wisdom of the Perl Monks concerning the following question:

I have been looking at utilizing Parallel::ForkManager in a project where I work, but I have run into a question for which I hope someone may be able to shed some light, or suggest a solution.

The project I am working on involves a process which processes information regarding a large number of devices, which means it may take several hours to collect and process all the information being used. I am looking at using P::FM as a way of handling a number of these concurrently to reduce the running time.

Because of the processing involved, however, there is a limited window of time in which the process may run without impacting other systems. Looking through the Perl Cookbook, I found example 16.21, referring to the use of eval() and signals, and thought perhaps I could combine the approaches. This is where my experience fails me as to what to expect.

Below is a snippet (haven't started writing the actual code) to give an idea of what I am thinking, but I want to confirm that there will be no weirdness that is immediately apparent, and that I am on the right track with the idea. Does this seem reasonable? Is there a better way? Is my understanding of what will happen with set_max_procs(0) correct?

Let me express my appreciation for any comments and assistance in advance.

#!/usr/bin/perl -w # # Sample snipped, containing code taken from the # documentation for Parallel::ForkManager and # example 16.21 from the _Perl_Cookbook_. # use strict; use vars qw($pm); use Parallel::ForkManager; # # Maximum of 20 simultaneous processes, # and begin graceful shutdown after 6 hrs. # my $MAX_PROCESSES = 20; my $MAX_TIME = 6 * 3600; $pm = new Parallel::ForkManager($MAX_PROCESSES) or die("Unable to create ForkManager object: $!\n"); # # &load_data() subroutine defined elsewhere # my (@all_data); @all_data = &load_data(); $SIG{ARLM} = sub { die("TimeOut"); }; eval { alarm($MAX_TIME); foreach my $data (@all_data) { my $pid = $pm->start and next; # Process or call processing routines # for $data here $pm->finish; } $pm->wait_all_children; alarm(0); }; if ($@) { if ($@ =~ m/TimeOut/) { $pm->set_max_procs(0); $pm->wait_all_children; } else { alarm(0); die; } }

Replies are listed 'Best First'.
Re: Graceful shutdowns and Parallel::ForkManager
by RMGir (Prior) on Jul 25, 2002 at 19:04 UTC
    I'm afraid that won't work for you.

    Here's the implementation of P::FM::start (found by doing vi `perldoc -l Parallel::ForkManager`):

    sub start { my ($s,$identification)=@_; die "Cannot start another process while you are in the child process +" if $s->{in_child}; while ( ( keys %{ $s->{processes} } ) >= $s->{max_proc}) { $s->on_wait; $s->wait_one_child(defined $s->{on_wait_period} ? &WNOHANG : undef +); }; $s->wait_children; if ($s->{max_proc}) { my $pid=fork(); die "Cannot fork: $!" if !defined $pid; if ($pid) { $s->{processes}->{$pid}=$identification; $s->on_start($pid,$identification); } else { $s->{in_child}=1 if !$pid; } return $pid; } else { $s->{processes}->{$$}=$identification; $s->on_start($$,$identification); return 0; # Simulating the child which returns 0 } }
    If you look at the "} else {" branch, you'll see that if no processes are available (when $s->{max_proc}==0, what set_max_procs(0) does), the task is continued in the parent process.

    That looks like a cool feature, actually, since you can test out your code in a single process loop before going parallel, if you like.

    I think you should have your alarm sig set a flag, which you could check in your "my $data" loop:

    my $stopFlag=0; $SIG{ARLM} = sub { $stopFlag=1; }; alarm($MAX_TIME); foreach my $data (@all_data) { last if $stopFlag; my $pid = $pm->start and next; # Process or call processing routines # for $data here $pm->finish; } $pm->wait_all_children; alarm(0);
    Hmmm, I think that'd work... Not sure, though.
    --
    Mike

      Thanks for commenting, RMGir-I greatly appreciate the input.

      After reading your comment, I felt a little more confident that a test would not bork my box. The test code is below, but the results seemed to indicate that it actually did work, surprisingly enough. (I get a list of numbers, the "Who's left?" message, then what appears to be those in the queue at the time of the alarm.) I'm not sure which parts I'm reading incorrectly in the module's code that make it appear that it shouldn't work, though. Can someone verify that they get the same/similar results?

      #!/usr/bin/perl -w use Parallel::ForkManager; use strict; use vars qw($pm); sub max_processes { 5; } sub max_seconds { ( ( ( ( ( 0 * 24 ) + 0 ) * 60 ) + 0 ) * 60 ) + 10; } # Time in seconds - 0d 0h 0m 10s $pm = new Parallel::ForkManager(&max_processes); $SIG{ALRM} = sub { die ("TimeOut"); }; eval { alarm( &max_seconds() ); foreach my $data (0..500) { my $pid = $pm->start and next; print($data); sleep(int(rand(10))); print('-', $data, "\n"); $pm->finish; } print("Waiting on everyone...\n"); $pm->wait_all_children; alarm(0); }; if ($@) { if ( $@ =~ m/TimeOut/ ) { # timed out; do what you will here $pm->set_max_procs(0); print("Who's left?\n"); $pm->wait_all_children; } else { alarm(0); die; } }
      Results:
      2-2
      3-3
      6-6
      5-5
      8-8
      4-4
      9-9
      7-7
      12-12
      0-0
      10-10
      1-1
      11-11
      Who's left?
      14-14
      15-15
      16-16
      13-13
      17-17
      
        just a quick comment...

        sub max_processes  { 5; }
        if you're trying to make this an inlined constant, you'll need to provide an empty prototype. i prefer making my constants ALL CAPS, and referring to them without a sigil. something like:

        sub MAX_PROCESSES() { 5 }

        and later...

        $pm = Parallel::ForkManager->new(MAX_PROCESSES);

        you'll find more info on this technique in perlsub.

        ~Particle *accelerates*

        D'oh. I completely missed that you were using the alarm to interrupt the eval.

        But since interrupting the eval kills your while loop, why bother with set_max_procs at all? No other children can be dispatched anyhow...
        --
        Mike