Fork Wait and probably grandchildren

Discordanian has asked for the wisdom of the Perl Monks concerning the following question:

I'm having some difficulty with forking and waiting for my processes to finish. At first I inserted a big 'sleep' after my wait, but after fiddling with the source I was able to duplicate my issue in a much smaller snippet of code. Bear with me:

#! /usr/bin/perl -w -I.

use strict;

######################################################################
+##########
# DATA
######################################################################
+##########

# Standard Modules
use Data::Dumper;

my $pid      = '';                                                    
+       # For forking.

my $templog    = '';

my @smokearr = ( 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h');

# Run ls over and over.
#---------------------------------------------------------------------
+-
for ( my $index = 0 ; $smokearr[$index] ; $index++ ) {
        $templog  = "./$smokearr[$index].log";
        my $syscmd = "./smoke.sh > $templog 2>&1";
        fail("fatal: cannot fork: $!") unless defined( $pid = fork() )
+;

        # Run sort as a child and exit
        if ( !$pid ) {
                print STDERR "Running [$syscmd] ($$)\n";
                system($syscmd) == 0
                  or print STDERR "fatal: system  \"$syscmd\" failed: 
+$? See log $templog\n";
                exit 0;
        }
}

# Wait for the child processes to finish.
#----------------------------------------
wait;

# Check each log file for smoke.pl
#---------------------------------------------------------------------
+---
for ( my $index = 0 ; $smokearr[$index] ; $index++ ) {
        $templog  = "./$smokearr[$index].log";
        my $syscmd = "grep smoke.pl $templog";

        print STDERR "Running [$syscmd] ($$)\n";
        system($syscmd) == 0
          or print STDERR "fatal: system  \"$syscmd\" failed: $? See l
+og $templog\n";
}

exit 0;
[download]

The contents of smoke.sh:

#!/bin/sh

cat smoke.txt |sort | uniq -c
[download]

As you can see, smoke.sh is trivial, but it has pipes.

The contents of smoke.txt is just a long text listing with smoke.pl being a line in there.

When I run my app, smoke.sh runs, wait gets the returned children but I get several failures in the log.

My theory is that smoke.sh returns before the piped grandchildren (I originally just had a cat without the pipes, but the error didn't occur). I also get the error if I put $syscmd = "cat smoke.txt |sort |uniq -c > $templog 2>&1"; directly.

So how can I get around this? I do a fork call 11 times writing to 11 log files and then I'm grepping those log files for a certain result and I get errors and this is presumably because the app I'm calling forks a writing thread.

$ uname -a
AIX nevd1 1 5 000DA5AF4C00 unknown unknown AIX
$ perl --version

This is perl, v5.6.0 built for aix
[download]

Any and all help is greatly appreciated.

Comment on Fork Wait and probably grandchildren Select or Download Code

Replies are listed 'Best First'.
Re: Fork Wait and probably grandchildren by polettix (Vicar) on Apr 05, 2005 at 15:39 UTC
You're surely not `wait`ing enough. You spawn 8 (not 11) child processes, and you have to `wait` for all of them: `1 while (wait > 0);` [download] Let me also note that you're cluttering your `@smokearr` doing the cycles the way you do. When you use the cycle the first time, it terminates after `$index` becomes equal to 8, adding an element `undef` to the array, which evaluates to false. Whether this is what you really wanted is up to you, of course, but I'd feel safer using something like: `for (my $index = 0; $index < @smokearr; ++$index) { #...` [download] This takes advantage from the fact that an array evaluated in scalar context gives you the number of elements in the array itself. Moreover, you don't seem to use `$index` other than for indexing the array, in which case I'd better use the more Perlish: `foreach my $item (@smokearr) { # ... use $item instead of $smokearr[$index] ... }` [download] which, in my opinion, improves readability (and it's less typing, too :). Flavio (perl -e "print(scalar(reverse('ti.xittelop@oivalf')))") Don't fool yourself.	[reply] [d/l] [select]
Re^2: Fork Wait and probably grandchildren by Discordanian (Initiate) on Apr 05, 2005 at 18:00 UTC
The while (wait > 0) did the trick so thanks muchly. The other suggestions are to an app where I was really just trying to reproduce the issue I was having witout posting 4,000 lines of code. In the application I have, I have 11 background processes (for this example only 8) and I do use foreach, just was typing quickly this morning and I'm a native 'C' speaker so it comes off the fingers quicker for quick-and-dirty things. (give me time and I'll be horking up my C code with foreach. :) )	[reply]
Re^2: Fork Wait and probably grandchildren by tlm (Prior) on Apr 05, 2005 at 17:19 UTC
frodo72, I've seen the `1 while wait > 0` idiom before, but I don't understand the rationale for it. It ends up calling `wait` at least once, maybe twice if all went well. But why the loop? Why not just collect the first value that `wait` returns if you care to know what it is, or else just call `wait` once in a void context? the lowliest monk	[reply] [d/l] [select]
Re^3: Fork Wait and probably grandchildren by polettix (Vicar) on Apr 05, 2005 at 17:52 UTC
Please consider that I've a Linux bias answering this kind of questions, so don't trust them too much :) `wait` basically waits until one of the children exits. It is used to avoid having Zombie processes: if you don't `wait()` for them, and you go a long time in your parent process before exiting, you'll see these processes in the process list even if they already exited, marked as "zombie" (this means that resources in the kernel associated to the process aren't freed). So, `wait` is the way by which the parent process acknowledges the kernel of the son's death. It's interesting that this mechanism actually tries to force the programmer not to ignore return values from child processes, much like the good practice of never throwing away return values from functions. Moreover, `wait` returns as soon as the first child exits; in the OP's post, (s)he created 8 of them, and had to wait until all of them have exited. According to `perldoc -f wait`, a call to the `wait` function returns -1 if, and only if, there are no more children alive(1) - thus the cycle. (1)Ok, ok, it's not exactly like this. If you happened to set `$SIG{'CHLD'}` to `'IGNORE'`, this zombie-ridding happens automatically, and you're likely to have `wait` always return -1 - but I guess it's not the case with this thread :) Moreover, in this case I would discourage the OP to do that, because (s)he's using wait as a synchronisation mechanism to understand when all the children exited. BTW, this is documented in `perldoc perlipc`, but I suggest taking a look in `perldoc perlport` as well. Flavio (perl -e "print(scalar(reverse('ti.xittelop@oivalf')))") Don't fool yourself.	[reply] [d/l] [select]
Re^4: Fork Wait and probably grandchildren by tlm (Prior) on Apr 05, 2005 at 18:07 UTC
Re^3: Fork Wait and probably grandchildren by RazorbladeBidet (Friar) on Apr 05, 2005 at 17:24 UTC
Because wait ... ...waits for a child process to terminate and returns the pid of the deceased process... So while child process A dies and it returns some PID > 0 child process B continues to run. It will return -1 when there are no child processes. Unlikely to happen, but since it is linked to wait(2) - a possibility, perhaps? -------------- "But what of all those sweet words you spoke in private?" "Oh that's just what we call pillow talk, baby, that's all."	[reply]
Re^4: Fork Wait and probably grandchildren by tlm (Prior) on Apr 05, 2005 at 17:33 UTC
Re^5: Fork Wait and probably grandchildren by RazorbladeBidet (Friar) on Apr 05, 2005 at 17:40 UTC
Re: Fork Wait and probably grandchildren by RazorbladeBidet (Friar) on Apr 05, 2005 at 17:00 UTC
Might also want to examine waitpid and check those return codes. I've had some interesting stuff happen with AIX and forking. -------------- "But what of all those sweet words you spoke in private?" "Oh that's just what we call pillow talk, baby, that's all."	[reply]
Re: Fork Wait and probably grandchildren by tlm (Prior) on Apr 05, 2005 at 16:34 UTC
Put `wait` inside the loop. the lowliest monk	[reply] [d/l]