http://qs1969.pair.com?node_id=288491

xiper has asked for the wisdom of the Perl Monks concerning the following question:

Well, that's about the best title i could think of...

I have a program that spawns a bunch of processes that send data back to the parent via a named pipe. This seemed to work fine until a came across a bug that, after many hours of reducing code to duplicate the problem, appears to have the following effect:

When the pipe contains a certain amount of (unread) data, usually around 10K, wait will hang indefinitely.

My code:

#! /usr/bin/perl use strict; use warnings; my $NUMLOOPS = 250; my $MAXPROCESS = 10; my $DATALEN = 100; my $VERBOSE = 1; # -------------------------------------- my( %pid ); pipe( CHILD, PARENT ); if(! $VERBOSE ) { close( STDERR ) } foreach my $num ( 1 .. $NUMLOOPS ) { print STDERR ">$num> processing\n"; if( keys( %pid ) >= $MAXPROCESS ) { print STDERR ">$num> waiting for reap\n"; &reap(); } if( my $pid = fork() ) { print STDERR ">$num> forked\n"; $pid{$pid} = $num; } else { close( CHILD ); print PARENT "x" x $DATALEN . "\n"; close( PARENT ); select( '', '', '', rand ); print STDERR ">$num> finished\n"; exit 0; } } while( %pid ) { &reap() } close( PARENT ); print "parent received: "; while( my $child = <CHILD> ) { print "."; } print "\n"; close( CHILD ); sub reap { my( $pid ) = wait(); if( $pid == -1 ) { die( "no more kids to reap!\n" ) } if(! $pid{$pid} ) { die( "why did I see $pid?\n" ) } print STDERR ">$pid{$pid}> reaped\n"; delete( $pid{$pid} ); }
On my system (v5.6.1, solaris), this code will produce the following:
...<snip>... >101> processing >101> waiting for reap >88> finished >88> reaped >101> forked >102> processing >102> waiting for reap >84> finished >84> reaped >102> forked >103> processing >103> waiting for reap *hangs*
If i increase $MAXPROCESS to 20, it hangs at 113 instead (10 more). If i double $DATALEN, it hangs at 57 (roughly half).

I don't claim to be an expert at multi-processes or ipc, although i would assume a pipe should be able to hold more than 10K at a time... Or is there something obvious i'm missing? Any help appreciated!

- ><iper

my JAPH: print"Just another Perl hacker\n"; # ^ look, no space! pretty tricky huh?

Replies are listed 'Best First'.
Re: Wait hangs when pipe is full
by Thelonius (Priest) on Sep 03, 2003 at 03:50 UTC
    I don't claim to be an expert at multi-processes or ipc, although i would assume a pipe should be able to hold more than 10K at a time... Or is there something obvious i'm missing? Any help appreciated!
    Nope. In fact, the maximum that's guaranteed on Solaris is 5120 bytes.
Re: Wait hangs when pipe is full
by Abigail-II (Bishop) on Sep 03, 2003 at 08:44 UTC
    The problem is that your buffers are filling up. You keep on writing to your pipe, but you don't read from it until all children have finished writing. But eventually, the buffer fills up, the child that's currently writing cannot write and hence will block. In the main time, your main program is waiting for the child to exit.

    So, you have a deadlock. You have a child that is waiting for the parent to read something, and a parent that's waiting for a child to terminate.

    Abigail

Re: Wait hangs when pipe is full
by Zaxo (Archbishop) on Sep 03, 2003 at 03:55 UTC

    The following rearrangement may help,

    # ... the loop close( PARENT ); print "parent received: "; print '.' while <CHILD>; print "\n"; close( CHILD ); while( %pid ) { reap() }
    That lets you draw down the buffer, making room for the other kids to print. It would also do no harm to try autoflushing the PARENT handle.

    After Compline,
    Zaxo

Re: Wait hangs when pipe is full
by asarih (Hermit) on Sep 03, 2003 at 03:40 UTC
    For one thing, this code doesn't use named pipes. Named pipes are ones that are created with mkfifo, for instance.

    For another, read up on pipe. It specifically warns you about deadlocks.

    The Camel has a few more words on this subject as well.

Re: Wait hangs when pipe is full
by xiper (Friar) on Sep 03, 2003 at 04:34 UTC
    Thanks all for your comments.

    asarih

  • "Named pipes" - my bad, slip of the tongue. I meant the opposite to perls 'anonymous pipes' (ie, open( "| ..." )).

    Thelonius++

  • 5120 bytes, okay, i didn't know that - presumably that's the problem then. I'll just create another process to read the data as its generated.

    Zaxo

  • I see your point, but no it doesn't, because the reading of <CHILD> still only occurs after the loop has finished. At best, it may free up space for the last $MAXPROCESS processes to write to.
  • Already tried the autoflushing, no luck.

    - ><iper

    my JAPH: print"Just another Perl hacker\n"; # ^ look, no space! pretty tricky huh?