Reading from Many Child Processes using Pipes

sved has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks, I am trying to write a script that:

1. Forks off multiple child processes (as few as 1, or as many as 1500)

2. Use pipes so all child processes communicate results periodically to the parent. (The child processes periodically (anywhere between 1 - 5 seconds) write some performance values as they go about doing their task)

3. Run a master loop in parent to poll all the reader pipes and (for now) print what is printed by the child processes. (Later I need to expand this to push all the data to a web application server as a JSON array. That is the easy part and not of concern at the moment)

(I read that using select is a better option than polling, but am unable to figure out how to use select in this context. A bonus thank you if you can help me with using select in the parent. One of the main hurdles for me is the fact that the child processes can run for quite sometime and having the parent wait until the child exits is impractical. The parent also needs to be reporting the values periodically.)

To begin with, I used a for loop to fork off child processes (after creating a pipe). Then use an infinite while loop to keep reading from the pipes. I know something is wrong, but need some guidance on it.

Here is the code:

#!/usr/bin/perl
use strict;
use warnings;
use IO::Handle;

my @kids = ();
my @readers;
my @writers;
my $pid;
foreach my $i (1..3) {
    pipe $readers[$i], $writers[$i] or die "Unable to Create read/writ
+e pipe for #$i\n";
    $writers[$i]->autoflush(1);
    $pid = fork;
    unless (defined($pid)) {
        print "Unable to fork a child\n";
        exit(1);
    } elsif ($pid) {    # Parent
        push @kids, $pid;
        close($writers[$i]);
        next;
    } else {            # Child
        print "Child Created: $$\n";
        close($readers[$i]);
        $| = 1;
        my $report;
        foreach my $i (1..10) {
            $report = sprintf("%.2f", rand(100));
            print $writers[$i],"[$$] $report\n";
            sleep(2);
        }
        exit(0);
    }
}
my $rin;
if ($pid) { 
    print "Kids: @kids\n";
    print "Readhing from Child handle\n";
    while (1) {
        foreach my $i (0..$#kids) {
            if (<$readers[$i]>) {
                if (defined($_)) {
                    print "[parent] $_\n";
                } else { 
                    print "[parent] <Empty>\n";
                }
            } else {
                print "Nothing to read\n";
            }
        }
        sleep(1);
    }
}
[download]

I know there are a few (unrelated to the problem) issues in the code, but in the interest of getting a working solution, I am ignoring them, on purpose.

Comment on Reading from Many Child Processes using Pipes Download Code

Replies are listed 'Best First'.
Re: Reading from Many Child Processes using Pipes by BrowserUk (Patriarch) on Oct 31, 2015 at 16:23 UTC
I know there are a few (unrelated to the problem) issues in the code, but in the interest of getting a working solution The major problem is that if one of your kids fails to write because its hung; then your entire sets of processes will block. The parent will block at the readline the first time it tries to read from the blocked child; and all the other children will block once they've written a buffer full of output to their pipe that the parent is no longer reading from. `while (1) { foreach my $i (0..$#kids) { if (<$readers[$i]>) {` [download] The solution is to use select which will prevent you from trying to read from a kid that hasn't written something for you to read. There are lots of select loop examples on perlmonks to get you started. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^2: Reading from Many Child Processes using Pipes by sved (Novice) on Oct 31, 2015 at 18:39 UTC
Thanks BrowserUK. The child processes are designed to always be non-blocking (or block with a timeout) and so will never hang. And, yes I was indeed looking for help with select. The select syntax is what confused me.	[reply]
Re^3: Reading from Many Child Processes using Pipes by BrowserUk (Patriarch) on Oct 31, 2015 at 20:18 UTC
will never hang. That's a brave assumption. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :) In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re: Reading from Many Child Processes using Pipes by Anonymous Monk on Oct 31, 2015 at 18:07 UTC
Slightly simplified and tweaked... #!/usr/bin/perl # http://perlmonks.org/?node_id=1146579 use IO::Select; use strict; use warnings; $\| = 1; my $children = 1000; my $sel = IO::Select->new; my $parentid = $$; for my $i (1..$children) { if( open my $fh, '-\|' ) { $sel->add($fh); } else # child { print "Child Created: $$\n"; for (1..10) { select undef, undef, undef, 2 + rand 0.1; printf "[%d] %5.2f\n", $$ - $parentid, rand 100; } exit; } } while( $sel->count ) { for my $fh ( $sel->can_read ) { if( sysread $fh, my $buf, 4096 ) { print "[parent] $buf"; } else { $sel->remove($fh); } } } print "[parent] all children have exited.\n"; [download] NOTE: check your limit on open files (ulimit -n) to see how many children you can have. Typical systems may have a limit of only 1024, but this might be tweakable.	[reply] [d/l]
Re^2: Reading from Many Child Processes using Pipes by sved (Novice) on Oct 31, 2015 at 18:48 UTC
Spot on AnonymousMonk. Thank you for giving me a much needed pointer. I'm curious why you used a select in the child code where I used sleep. Is there an advantage of using select over sleep? Or was it just for simulating random sleep for this example?	[reply]
Re^3: Reading from Many Child Processes using Pipes by Anonymous Monk on Oct 31, 2015 at 19:22 UTC
From perldoc -f sleep : "sleep Causes the script to sleep for (integer) EXPR seconds" I wanted fractional second sleeps, which select provides, without having to use Time::HiRes	[reply]