I'm using fork() and having trouble limiting the number of child processes that spawn. Say I have 12 tasks to do in a list. My technique is to to set a batch size ($children = 5) and process the list one by one, counting as I go along. If I've reached the batch size ($counter mod $children becomes false) or I'm at the end of the list, then go ahead and execute on this batch. Executing involves forking out $children number of child processes that take 15-20 seconds each to complete.

The problem is that even though the logical order of my code says to: 1) create the batch, then 2) do something with this batch, then 3) create another batch, it isn't working in that order. Instead, I watch as 12 child processes go out and do their thing. Even though part of the duty of each process is to print data and that data has not yet been returned to me, the loop figures it's done and proceeds to the next batch. I don't get it.

Here's an example - shouldn't this snippet send off 5 processes which sleep for 15 seconds, then print 5 values, after which another 15 seconds passes and another 5 values are printed, then 15 seconds and the last two are printed? Instead, it sleeps for 15 seconds and prints all 12 values. 

#!/usr/bin/perl
# batch.pl

use IO::Select;
use IO::Pipe;

my @list = ("a", "b", "c", "d", "e", "f", 
            "g", "h", "i", "j", "k", "l");
my $children = 5;
my $counter = 0;
my $size = @list;

foreach (@list) {
  $counter++;
  if (($counter%$children) && ($counter != $size)) {
    push(@temp,$_);
  } else {
    push(@temp,$_);
    @batch = @temp;
    @temp = @blank;
    foreach (@batch) {
      my $pipe = IO::Pipe->new();
      my $pid = fork;
      die "Bad Fork $!" unless defined $pid;

      if ($pid) {
        $pipe->reader();
#        $select->add( $pipe );

      } else {
        $pipe->writer();
        $pipe->autoflush(1);

        &dosomething;
        exit;
      }
    }
  }
}

sub dosomething {
  sleep 15;
  print "$_\n";
}

[download]

By the way, I tried using waitpid() with no success. Perhaps I was using it incorrectly. Is waitpid() the way to go, or is my logic just totally wrong?

In reply to how to limit child processes? by Galen

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.