Please help me fork my 'children'...(that went to a dark place)..

My problem: Every day - we would like to run ~140 backup commands that our DBA's have come up with.

(This is under Linux)

The problem is that we cannot run all backup commands at the same time or it would clobber our performance. We do not really know one day to the next what commands will take 2 hours and what commands will take 10 minutes. So we cannot do batches.

My plan: Take an array of commands and fork each one to a child process up to 6. Every so often - see if any finished and fire up the next command. This keeps things running, but keeps things from running away.

The down side - I have never done this type of thing in perl. I've done it with bash scripts but Perl should be able to do a better job - right?

An afternoon of study allowed me to come up with the following test code that seems to work. But I need advice from people more experienced to look at things and tell me how to head off problems I dont know about yet.

Question: If a command fails, does "waitpid" put the exit code into the $? variable or how do I get the exit code of a command after it finishes?

The code below can be cut and pasted:

use POSIX ":sys_wait_h";
...
#---------------------------------------------------------------------
+----------------------------------------
#---------------------------------------------------------------------
+----------------------------------------
sub S_TestForking2 {

    my ($lSleepTime);
    my ($lProcessLabel, @lPidArray);
    my $lFinished = 0;
    my $lStillRunning = 0;
    my $isAlive;
    my $kid;
    my $lpid;
    my $i;
    
    my %lRunningPidHash = ();

    my @lCmdArray;
    my $lCmdIndex;

    # Create a bunch of fake commands that take time
    push (@lCmdArray, "sleep 20");
    push (@lCmdArray, "sleep 25");
    push (@lCmdArray, "sleep 38");
    push (@lCmdArray, "sleep 20");
    push (@lCmdArray, "sleep 17");
    push (@lCmdArray, "sleep 15");
    push (@lCmdArray, "sleep 25");
    push (@lCmdArray, "sleep 30");
    push (@lCmdArray, "sleep 15");


    $lFinished = 0;
    while ($lFinished == 0) {
        
        # Count the running processes using the pid array we filled wh
+en we forked the child

        $lStillRunning = 0;
        foreach $lpid (sort keys %lRunningPidHash) {
            waitpid($lpid, WNOHANG);
            if ( $? == -1 ) {
                $lStillRunning++;
            } elsif ($? == 0) {
                # If we ask about this id again  we get -1 forever
                delete $lRunningPidHash{$lpid};
            }
            print "\t$lpid = $?\n";
        }

        print "Running processes: $lStillRunning\n";

        # If our command array is zero and our running processes is ze
+ro - we can just exit

        if ( $lStillRunning == 0 and scalar (@lCmdArray) == 0 ) {
            $lFinished = 1;
            last;
        }


        # Pop the commands off the command array so we always have 6 p
+rocesses running

        for ($i = $lStillRunning; $i < 3; $i++) {
            my $lCmd = pop (@lCmdArray);
            if ( $lCmd ) {
                $lpid = fork;
                die "Couldn't fork: $!" unless defined $lpid;

                if ( $lpid ) {
                    # parent
                    print "Forked off a child: $lCmd - $lpid\n";
                    $lRunningPidHash{$lpid} = 1;
                } else {
                    # child
                    close (STDIN);
                    close (STDOUT);
                    close (STDERR);
        
                    exec ($lCmd) or die ("Error: could not exec comman
+d : $lCmd :  $!\n");
                }
            }
        }

        # Sleep a few seconds and re scan
        sleep (5);
                  
    }



    print "Parent exiting\n";


}   # S_TestForking2
[download]