waavman has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have written a perl script in which the parent spawns a child process using perl Pipes

$pid = open(PARENT_FILE_HANDLE,"-|");

Child does some stuff like ssh to a remote machine and gets some info which it then prints to STDOUT. The parent then reads the information from the PARENT_FILE_HANDLE.

The problem is that when the child tries to print a lot of data to STDOUT, the child just hangs after printing a few lines.

To simulate this problem in a simpler way , I have written an equivalent script below in which the child reads data from a file "/tmp/testfile" which contains around 200 lines and then writes the lines to STDOUT which the parent process then tries to read from PARENT_FILE_HANDLE. But the child process just hangs after printing a few lines to STDOUT and the output from the code below shows that the parent process waits for 30 seconds and then kills the hung child process.

The script works fine only if /tmp/testfile has fewer than 50 lines.

Please let me know why this could be happening and how to counter this problem


#!/usr/bin/perl
#fork a process
$pid=open(PARENT_READ_HANDLE, "-|");
if ($pid==0) # child process
{
open (CHILDREADHANDLE, "</tmp/testfile");
while (<CHILDREADHANDLE>)
{
($sec,$min,$hour) = (localtime) 0,1,2;
print "$hour:$min:$sec - $_ \n";
}
close(CHILDREADHANDLE);
exit(0);
}
else #parent process
{
# thIs section checks and makes sure child process if Terminated
# if its hung for more than 30 seconds.
$count =0;
$childprocess = qx (ps -ef|grep -v defunct|grep -v grep |grep $pid);
while (($childprocess ne "") && ($count < 6))
{
$count+=1;
printf ("Found child process still running count=$count\n");
sleep (5);
$childprocess = qx (ps -ef|grep -v defunct|grep -v grep |grep $pid);
}
if ($childprocess ne "")
{
qx (kill -9 $pid);
printf (STDERR "Child ssh process hung. So forcibly killed it\n");
}
waitpid($pid,0);
# read the data printed by child
while (<PARENT_READ_HANDLE>)
{
printf ("Parent received from the child : $_\n");
}
}
exit(0);



-waavman
  • Comment on Issue with communication of large data between Parent and child using Perl pipes

Replies are listed 'Best First'.
Re: Issue with communication of large data between Parent and child using Perl pipes
by ikegami (Patriarch) on Jul 17, 2009 at 21:45 UTC

    Pipes can only hold a limited number of bytes, and your pipe is getting filled.

    • Your child is waiting for the parent to start emptying the pipe so it can put more into it.
    • Your parent is waiting for the child to end.

    You have a Deadlock.

    Run the parent asynchronously instead of waiting for the child to end, or use qx// to launch the child.

    By the way, it's probably not a good idea to use waitpid on a child launched using open. Closing the handle is the appropriate way of reaping the child.

    By the way, please put computer text (code, data) in between <c>...</c> tags. It preserves the formatting, escapes characters that need escaping and provides a handy download link.

      Thanks Ikegami. Your explanation of the concept that Pipes can hold only a limited number of bytes and how the deadlock is occurring made things clear. So now I made the parent read operation asynchronous by moving the read operation by the parent process BEFORE it checks to see if the child process has terminated. After it kills the child, I have made it read one last time again just in case child wrote something to the file handle PARENT_READ_HANDLE just before being killed.
      After making these changes, the code runs without any issues now
      else #parent process { # This is the section I added before checking for # child's status and it solved the problem while (<PARENT_READ_HANDLE>) { printf ("Parent received from the child : $_\n"); } # thIs section checks and makes sure child process if Terminated # if its hung for more than 30 seconds. $count =0; $childprocess = qx (ps -ef|grep -v defunct|grep -v grep |grep $ +pid); while (($childprocess ne "") && ($count < 6)) { $count+=1; printf ("Found child process still running count=$count\n") +; sleep (5); $childprocess = qx (ps -ef|grep -v defunct|grep -v grep |gr +ep $pid); } if ($childprocess ne "") { qx (kill -9 $pid); printf (STDERR "Child ssh process hung. So forcibly killed + it\n"); } waitpid($pid,0); # read the data printed by child while (<PARENT_READ_HANDLE>) { printf ("Parent received from the child : $_\n"); } }

      As a note, there is documentation on opening pipes using "-|" at the below URL under the section "Safe Pipe Opens"
      http://www.perl.com/doc/manual/html/pod/perlipc.html

      thanks waavman
Re: Issue with communication of large data between Parent and child using Perl pipes
by mzedeler (Pilgrim) on Jul 17, 2009 at 21:03 UTC

    Please use <code> to get your code properly formatted. You also should use

    use warnings; use strict;

    In the top of your script.

    That aside, did you read perlipc?

Re: Issue with communication of large data between Parent and child using Perl pipes
by Marshall (Canon) on Jul 17, 2009 at 21:12 UTC
    First, I would always recommend the use of warnings and strict;

    What you have here doesn't appear to be a client-server application. For example: $pid=open(PARENT_READ_HANDLE, "-|"); "open()" does NOT return a PID, it returns status code from the open. The open function does not create a PID. For something like this:

    open (PROG, "some_program its_args |") or die "can not run some_program $!";
    open creates a filehandle status return code! PROG is a filehandle that captures the output of "some_program". You can do: while(<PROG>){...blah...}. This is a synchronous open and is a way of allowing the main program to capture the output of "some_program". Both programs really aren't "running at the same time".

    I looked for that the "-" modifier does in front of pipe symbol, eg "-|" instead of just "|" and didn't figure that out. I doubt that the answer changes the above, but would be curious as to what that "-" means...thanks to anybody who can clue me in!

      I looked for that the "-" modifier does in front of pipe symbol,

      It's the 3-arg form of "command|"

      open(my $pipe, "$command |") # 2-arg form open(my $pipe, '-|', $command) # 3-arg form

      "open()" does NOT return a PID

      Actually, it does when using open to create a process and pipe. This is documented.

      And it appears that -| and |- without further arguments simply forks without executing anything instead of giving an error or trying to launch a program named "-". This doesn't seem to be documented. [ Apparently it is, just not in open. See reply ]

        This doesn't seem to be documented.

        It's documented in Safe Pipe Opens, with examples that might be helpful to the OP.

        Thanks! ikegami!

        open(my $pipe, "$command |") # 2-arg form open(my $pipe, '-|', $command) # 3-arg form
        above looks right, this 2 arg form from OP looked weird to me:
        $pid=open(PARENT_READ_HANDLE, "-|");
        Which as they say is neither fish or fowl (not correct 2-arg or 3 arg). I am still a bit confused as to the OP's intent here. This still appears to be a synchronous app, not client-server. There is a lot of complex stuff that doesn't appear to be necessary.
Re: Issue with communication of large data between Parent and child using Perl pipes
by jdrago_999 (Hermit) on Jul 21, 2009 at 17:17 UTC

    As others have noted, always start your Perl scripts with the following:

    #!/usr/bin/perl -w use strict; use warnings 'all';

    It can help prevent many bugs that otherwise would not have been obvious.