nikhil150585 has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have around 2000 system calls to execute. I want to execute 50 of these parallely at once and then wait on any of these to finish and then start the next one. In a way, I should see 50 system calls executing on my system at any given time.
I have tried using the fork spawning 8 process, each process executing one system call on my local machine at once and then use wait() function to wait on any of these process to get completed before forking/spawning the next one. This worked fine for me, but now I am running the same on the grid of machines where I am trying to fork 50 parallel process, and I am seeing now that more than one process can complete at a single time. so if let's say two process complete at the same time, I come out of wait() and fork a single process.
I am seeing that the number of process that I am executing at some later point of time is less than 50. Can you please help me, to keep the number of process/system calls constant to what I define.
Thanks, Nikhil

Replies are listed 'Best First'.
Re: Parallel system calls
by onelesd (Pilgrim) on Sep 26, 2011 at 22:15 UTC

    Parallel::ForkManager was written to do precisely this, and works very well.

      use Parallel::ForkManager;

  $pm = new Parallel::ForkManager($MAX_PROCESSES);

  foreach $data (@all_data) {
    # Forks and returns the pid for the child:
    my $pid = $pm->start and next;

    ... do some work with $data in the child process ...

    $pm->finish; # Terminates the child process
  }

    [download]
[reply]
[d/l]
[reply]
Re: Parallel system calls
by salva (Canon) on Sep 27, 2011 at 06:34 UTC
    using Proc::Queue: 
    use Proc::Queue size => 50,
                ignore_children => 1,
                qw(system_back);

for my $cmd (@cmd) {
  system_back $cmd;
}

    [download]
[reply]
[d/l]
Re: Parallel system calls
by Anonymous Monk on Sep 26, 2011 at 22:15 UTC
    If a signal has been issued to notify a process that a child has died, any number of children could die before the process wakes up and notifies the signal. A signal is a FLAG.
[reply]
Re: Parallel system calls
by thargas (Deacon) on Sep 27, 2011 at 11:56 UTC
    If you want to do everything yourself, you need to look at the waitpid builtin, see perldoc -f waitpid
[reply]
[d/l]
Re: Parallel system calls
by locked_user sundialsvc4 (Abbot) on Sep 27, 2011 at 13:36 UTC

    I would suggest using a combination of Parallel::ForkManager and a queue.

    Use the fork-manager to spawn a certain number of processes, then let each of them consume work-requests from a queue.   Or, if the requirement is simpler, let them decrement a shared variable and do so until the count reaches zero.

    In general, “the number of units-of-work that are to be performed,” and “the number of parallel workers who are tasked to do the work,” are and should always remain separate.   The number of workers controls what IBM used to call the multiprogramming level of the system, and that “knob” should be independent of the workload volume.

Back to Seekers of Perl Wisdom