TeKk9 has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to write a script that'll daemonize and allow no more than 6 children to fork but also keep 6 children running while there's work to be done. I'm processing some large data files that get inserted into a db and I've found that six is the max number I can comfortably process at once (I'm continually receiving them around 400/day), eventhough there maybe 20 or more in queue to process. Basically here's the code I have. As is I end up with zombies, but if I use a reaper function I end up getting a -1 back from wait. Any ideas/examples would be greatly appreciated. 
while(1) {

    $processes = 0;

    foreach $compressedFile ( @pdfFiles ) {

        $pdfFile = $compressedFile;

        $pdfFile =~ s/\.Z//;

# - For each pdf file fork a new process upto MAX_PROCESSES.

        if( $my_pid = fork()) {
            $processes++;
            if( $processes >= $MAX_PROCESSES ) {
                $wait = wait();
                print "WAIT => $wait\n" if( $debug );
                print "PID => $my_pid\n" if( $debug );
                if( $wait ) { $processes-- ; }
            } elsif ( !defined $my_pid ) {
                die "Error: Cannot fork process: $!";
            }
        } elsif ( $my_pid == 0 ) {
            &load_file($pdfFile);
            exit 0;
        }

    }

    if( $daemon_mode ) {
        $sleep = $sleepTime;
        print "LOAD_PDF => Sleeping $sleep seconds\n" if( $debug );
        sleep $sleep;
    } else {
        print "LOAD_PDF => Finished!\n" if ($debug);
        exit 0;
    }

}

[download]

Replies are listed 'Best First'.
Re: Daemonized with max processes
by Aristotle (Chancellor) on Aug 14, 2003 at 21:24 UTC
[reply]
•Re: Daemonized with max processes
by merlyn (Sage) on Aug 14, 2003 at 22:19 UTC
[reply]
Re: Daemonized with max processes
by dtr (Scribe) on Aug 14, 2003 at 22:04 UTC

    Perhaps it's just me, but supposing you give this program 5 files to work on, then you'll never actually execute the wait, since $processes >= $MAX_PROCESSES will never be true. This is probably why you're getting Zombies.

    I am no expert on this front, but if I were you, I'd try maintaining a global "child count" variable, and have a $SIG{CHLD} handler that decremented this value. Then, in your loop, you can check to see if there are any spaces to create another free child, and if so then do the fork.

[reply]
[d/l]
[select]
Re: Daemonized with max processes
by Marcello (Hermit) on Aug 15, 2003 at 09:38 UTC
    Hi,

    You should use a reaper, the following snippets should get you started: 
    use POSIX;

use constant MAX_CHILDS => 6;

my $CHILD_COUNT = 0;

$SIG{CHLD} = \&Reaper;

sub Reaper {
    my $childPid;
    while (($childPid = waitpid(-1, &WNOHANG)) > 0) {
        $CHILD_COUNT--;
    }
    $SIG{CHLD} = \&Reaper;
}

if ($CHILD_COUNT < MAX_CHILDS) {
    my $pid;
    if (!defined($pid = fork())) {
        # Fork failed
    }
    # Child process
    elsif ($pid == 0) {
        # Process file
    }
    # Parent process
    else {
        $CHILD_COUNT++;
    }
}

    [download]
[reply]
[d/l]
Re: Daemonized with max processes
by jmanning2k (Pilgrim) on Aug 15, 2003 at 14:25 UTC
    I like the code from SpamAssassin, and have borrowed it for many little projects of my own. Just remember, it's GPL, so if you release your program you have to GPL it too - but you are free to use it internally however you like.

    (UPDATE:I just noticed it's dual licensed with the Perl Artistic License, so it's "free-er" than I thought...)

    See the function start_children in this module. It uses IO::Socket to communicate with each child process.

    The reap_children function is also useful.

    ~Jon

[reply]

Back to Seekers of Perl Wisdom