Amphiaraus has asked for the wisdom of the Perl Monks concerning the following question:

I am having a problem with the www.cpan.org module Proc::Queue

Proc::Queue is successfully capping the maximum number of active child processes my program can have at 5, using the following code:

use Proc::Queue size => 5, debug => 1;

use POSIX ":sys_wait_h"; # imports WNOHANG

foreach (1..20) {
  my $f=fork;

 (my child processes do stuff)

  exit 0;  # stop a child process

  1 while waitpid(-1, WNOHANG)>0; # reaps children
}

[download]
My problem is, I have several functions in my Perl script that used forked processes, and I need to use Proc::Queue in each of these functions to cap the maximum number of active forked processes allowed.

However, I am finding that even before an earlier function in my main completes all its child processes, the next function called by my main starts, and begins to make forked processes of its own.

Is there a way to insure that a given function that uses fork() and Proc::Queue reaps all of its child processes, before execution of the script returns to main and starts the next function call in main?

The functions called in my Perl script's main have dependencies such that a given function MUST be 100% done with its activities, before the next function in main is called.

Thank You,
Amphiaraus

Replies are listed 'Best First'.
Re: Proc::Queue problem
by salva (Canon) on Oct 23, 2008 at 08:57 UTC
    You don't want to parallelize your main tasks so, don't fork on your main loop and ensure that no children are running at the end of every iteration: 
    use Proc::Queue size => 5;

for (1..20) {
  # you don't want to fork here!!!
  do_task($_);
  1 while wait != -1;
}

    [download]
[reply]
[d/l]

      I took it that he wanted to parallize several sub-tasks, like

      foreach (@subtask1) {
    fork;
    # do subtask1
}

wait; # for all subtasks to finish

foreach (@subtask2) {
    fork;
    # do subtask2
}

wait; # for all subtasks to finish

repeat for subtaskX

      [download]
      l8rZ,
      --
      andrew
[reply]
[d/l]
Re: Proc::Queue problem
by afresh1 (Hermit) on Oct 23, 2008 at 03:37 UTC

    I think you need to do a blocking wait for the children that you fork. It seems to work this way for me, but if you fork lots of processes, you may want to clean up some of the zombies as you go.

    I modified this from the example in the Proc::Queue docs

    #!/usr/bin/perl
use strict;
use warnings;

use Proc::Queue 'waitpids', size => 4, debug => 1;

my @pids;
foreach ( 1 .. 10 ) {
    my $f = fork;
    if ( defined($f) and $f == 0 ) {
        print "-- I'm a forked process $$\n";
        sleep rand 5;
        print "-- I'm tired, going away $$\n";
        exit(0);
    }
    push @pids, $f if defined $f;
}

waitpids(@pids);

print "--- FINISHED\n";

    [download]
    l8rZ,
    --
    andrew
[reply]
[d/l]

Back to Seekers of Perl Wisdom