Please help me fork my 'children'...(that went to a dark place)..

My problem: Every day - we would like to run ~140 backup commands that our DBA's have come up with.

(This is under Linux)

The problem is that we cannot run all backup commands at the same time or it would clobber our performance. We do not really know one day to the next what commands will take 2 hours and what commands will take 10 minutes. So we cannot do batches.

My plan: Take an array of commands and fork each one to a child process up to 6. Every so often - see if any finished and fire up the next command. This keeps things running, but keeps things from running away.

The down side - I have never done this type of thing in perl. I've done it with bash scripts but Perl should be able to do a better job - right?

An afternoon of study allowed me to come up with the following test code that seems to work. But I need advice from people more experienced to look at things and tell me how to head off problems I dont know about yet.

Question: If a command fails, does "waitpid" put the exit code into the $? variable or how do I get the exit code of a command after it finishes?

The code below can be cut and pasted: 


use POSIX ":sys_wait_h";
...
#---------------------------------------------------------------------
+----------------------------------------
#---------------------------------------------------------------------
+----------------------------------------
sub S_TestForking2 {

    my ($lSleepTime);
    my ($lProcessLabel, @lPidArray);
    my $lFinished = 0;
    my $lStillRunning = 0;
    my $isAlive;
    my $kid;
    my $lpid;
    my $i;
    
    my %lRunningPidHash = ();

    my @lCmdArray;
    my $lCmdIndex;

    # Create a bunch of fake commands that take time
    push (@lCmdArray, "sleep 20");
    push (@lCmdArray, "sleep 25");
    push (@lCmdArray, "sleep 38");
    push (@lCmdArray, "sleep 20");
    push (@lCmdArray, "sleep 17");
    push (@lCmdArray, "sleep 15");
    push (@lCmdArray, "sleep 25");
    push (@lCmdArray, "sleep 30");
    push (@lCmdArray, "sleep 15");


    $lFinished = 0;
    while ($lFinished == 0) {
        
        # Count the running processes using the pid array we filled wh
+en we forked the child

        $lStillRunning = 0;
        foreach $lpid (sort keys %lRunningPidHash) {
            waitpid($lpid, WNOHANG);
            if ( $? == -1 ) {
                $lStillRunning++;
            } elsif ($? == 0) {
                # If we ask about this id again  we get -1 forever
                delete $lRunningPidHash{$lpid};
            }
            print "\t$lpid = $?\n";
        }

        print "Running processes: $lStillRunning\n";

        # If our command array is zero and our running processes is ze
+ro - we can just exit

        if ( $lStillRunning == 0 and scalar (@lCmdArray) == 0 ) {
            $lFinished = 1;
            last;
        }


        # Pop the commands off the command array so we always have 6 p
+rocesses running

        for ($i = $lStillRunning; $i < 3; $i++) {
            my $lCmd = pop (@lCmdArray);
            if ( $lCmd ) {
                $lpid = fork;
                die "Couldn't fork: $!" unless defined $lpid;

                if ( $lpid ) {
                    # parent
                    print "Forked off a child: $lCmd - $lpid\n";
                    $lRunningPidHash{$lpid} = 1;
                } else {
                    # child
                    close (STDIN);
                    close (STDOUT);
                    close (STDERR);
        
                    exec ($lCmd) or die ("Error: could not exec comman
+d : $lCmd :  $!\n");
                }
            }
        }

        # Sleep a few seconds and re scan
        sleep (5);
                  
    }



    print "Parent exiting\n";


}   # S_TestForking2

[download]

In reply to Using perl to manage child processes - critique my code. by FatDog

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.