Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear wise monks,

I am trying to communicate with several child processes, each of which calls a web service and then writes a small amount of data on STDOUT. I want to read from all of them that I can, and then declare the rest timed out. The problem is that the open2 and IO::Select don't seem to be reliable.

I have trimmed the problem down to a single-child case. Sometimes I am able to read the process output, and sometimes it times out with nothing. I've even seen the child's output appear on the regular STDOUT as if it had never been redirected, while the parent process gets an empty buffer back!

Do you have any ideas about what could be going wrong, or suggestions for better ways to do this?

A code example is below.

use strict; use IPC::Open2; use IO::Select; sub sys_read { my ($fh) = @_; my $blksize = 4096; my $len; my $data = ""; my $buff; READS: while ($len = sysread($fh, $buff, $blksize)) { if (!defined $len) { next READS if $! =~ /^Interrupted/; return undef; } $data .= $buff; } return $data; } # The tester.pl script calls a web service using lines like this: #my $ua = LWP::UserAgent->new; #my $response = $ua->request(POST $uri, [ackhost => $ackhost, # data => $data, # pid => $pid]); my @command = ("/usr/bin/perl", "./tester.pl"); my($wtr, $rdr); # trying an autovivification this time. my $pid = open2($rdr, $wtr, @command); close $wtr; my @lst; push @lst, $rdr; my $timeout = 4.0; my @ready; READLOOP: while(@ready = IO::Select->new(@lst)->can_read($timeout)) { HANDLELOOP: foreach my $rh (@ready) { # This might block and sysread might be better, but let's see +.. my $buff = sys_read($rh); print "buff is *$buff*\n"; close $rh; @lst = (); } } if (scalar(@lst)) { print "Timed out with nothing this time\n"; }

Replies are listed 'Best First'.
Re: open2 unreliable?
by fglock (Vicar) on Nov 11, 2002 at 18:53 UTC

    You might want to turn off i/o buffering, so that you get anything the child writes, even if it times out.

Re: open2 unreliable?
by IlyaM (Parson) on Nov 11, 2002 at 23:28 UTC
    You can try to use IPC::Run. Very clean and flexiable API (I also prefer to use it over system for this reason) and it is more reliable (I've seen many people to complain about problems with IPC::Open2 under mod_perl which were fixed in no time when they switched to IPC::Run).

    --
    Ilya Martynov, ilya@iponweb.net
    CTO IPonWEB (UK) Ltd
    Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net
    Personal website - http://martynov.org

Re: open2 unreliable?
by MZSanford (Curate) on Nov 11, 2002 at 18:54 UTC
    does tester.pl buffer ? I am thinking that $| needs to be set to non-0 in tester.pl, thus flushing the buffer from it.
    from the frivolous to the serious
Re: open2 unreliable?
by Helter (Chaplain) on Nov 12, 2002 at 18:26 UTC
    I had to help someone fix an issue with IPC:Open2 a week or so ago, we were seeing all the processes turn to zombies in a Linux environment.

    Our problem was 2 fold, we were not flushing the buffers enough:
    $| = 1;
    This fixed that problem (buffer by line instead of chunk).

    The second problem was a little stranger (zombies invading our machines...late night sci-fi here we come!)

    I had to read 3 docs on IPC:Open2 before I found this on the CPAN site (IPC:Open2):
    open2() does not wait for and reap the child process after it exits. E +xcept for short programs where it's acceptable to let the operating s +ystem take care of this, you need to do this yourself. This is normal +ly as simple as calling waitpid $pid, 0 when you're done with the pro +cess. Failing to do this can result in an accumulation of defunct or +"zombie" processes. See perlfunc/waitpid for more information.
    You won't find this text (unfortunitly) in  perldoc perlipc. Or in the Camel, at least I can't find it :)

    Hope this helps, and I'm not making some glaring error :)