All - I found some code for a forking queue that I improved upon however I found a glitch in my logic. My script is going to eventually launch off backups for certain systems, with a queue limit of only 3 at a time. My problem is this: my script will hang at the wait, and even if I have a max computing time it will wait for a child to exit before proceeding. If my first few systems in the queue hang I will never get by this point in my script. (See below). Is there another way to loop through my child processess without using wait possibly (or possibly looking for a signal that one of the children exits). Any help would be greatly appreciated. 
#!/usr/local/bin/perl

my %children; # stores pid for child processes
my %processing_time; #stores processing time for child
my %child_output;
my $max_children = 3; # max queue size
my @queue;


foreach my $dir(qw/50 50 50 10 18 600 12 15/)
{
    fork_backup($dir);
}
complete_backup();


sub fork_backup
{
    my $lpar = shift;
    #print "Popped off $dir from queue\n";
    if ( ( scalar keys %children ) < $max_children )
    {
        my $pid = fork;
        if ( $pid )
        {
            #   this is parent process
            $children{$pid} = $lpar;
             print "Child Backup [$pid][".$children{$pid}."] started a
+t ".localt
ime()."\n";
            $processing_time{$pid} = time();
        }
        else
        {
            if ( not defined $pid )
            {
                die "\n\nWoah: Failed to fork a child!\n";
            }
            #   this is child process
            $child_output{$lpar}=`sleep $lpar`;
            #   exit child process
            exit 0;
        }
    }
    else
    {
        #   too many children queue for later
        unshift(@queue,$lpar);
    }
}

sub complete_backup
{
  while (%children)
  {
    # Lets check and see if anything is hanging around too long
    while ( my ($key, $value) = each(%processing_time) )
    {
      my $nowtime=time();
      if ( ($nowtime-$value) > 20)
      {
        print "Child Backup [$key][".$children{$key}."] will be killed
+.\n";
        kill($key);
        delete $processing_time{$key};
        delete $children{$key};
        next;
      }
    }
    my $childpid = wait();
    my $time=localtime();
    my $ptemp=$processing_time{$childpid};
    $ptemp=(time())-$ptemp;
    print "Child Backup [$childpid][" . $children{$childpid}. "] compl
+eted at $t
ime taking $ptemp seconds\n";
    delete $children{$childpid};
    delete $processing_time{$childpid};

    if ( @queue )
    {
      # Get the next item from the queue
      my $next_lpar= pop @queue;
      fork_backup($next_lpar);
    }
  }
}



Here is my output (it hangs on the first three "50" sleeps, until one 
+of the sleeps finishes even though my max timeout is 20):

stealth% ./fqueue
Child Backup [29211][50] started at Fri Jul 25 15:41:55 2008
Child Backup [29212][50] started at Fri Jul 25 15:41:55 2008
Child Backup [29215][50] started at Fri Jul 25 15:41:55 2008
Child Backup [29211][50] completed at Fri Jul 25 15:42:45 2008 taking 
+50 seconds
Child Backup [29325][10] started at Fri Jul 25 15:42:45 2008
Child Backup [29215][50] will be killed.
Child Backup [29212][50] will be killed.
Child Backup [29212][] completed at Fri Jul 25 15:42:45 2008 taking 12
+17018565 seconds
Child Backup [29326][18] started at Fri Jul 25 15:42:45 2008
Child Backup [29215][] completed at Fri Jul 25 15:42:45 2008 taking 12
+17018565 seconds
Child Backup [29329][600] started at Fri Jul 25 15:42:45 2008
Child Backup [29325][10] completed at Fri Jul 25 15:42:55 2008 taking 
+10 seconds
Child Backup [29331][12] started at Fri Jul 25 15:42:55 2008
Child Backup [29326][18] completed at Fri Jul 25 15:43:03 2008 taking 
+18 seconds
Child Backup [29338][15] started at Fri Jul 25 15:43:03 2008

[download]

In reply to Forking Queue Problem by smithgw

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.