morpheus has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,

I want to launch a programm, parse STDOUT and STDERR an react on different outputs. Unfortunately I cannot get rid of buffering. Here is a simple example

Program I want to launch:

#!/usr/bin/perl print "Please press return\n"; my $line = <STDIN>; print "OK, exit now\n";

Main program

#!/usr/bin/perl use strict; use warnings; use IO::Handle; use IPC::Open3; my $fh_out = new IO::Handle; my $fh_in = new IO::Handle; my $fh_err = new IO::Handle; my $pid = open3($fh_in,$fh_out,$fh_err,'./skript_above.pl'); $fh_out->autoflush(1); #Putting these lines above open3 $fh_err->autoflush(1); #doesn't seem to work either while (!$fh_out->eof) { my $line = $fh_out->getline; print "TIME: ".localtime()." OUTPUT: ".$line; if ($line =~ m/^Please press return/ ) { $fh_in->print("\n"); #provide the desired input } } waitpid($pid,0);

In the real program I use IO::Select to read $fh_out and $fh_err and the called program is a binary file (sqlplus)

Problem: The second script justs hangs because the output "Please press return\n" from the first script seems to be buffered. As a result the first script is already waiting for some input but never gets one.

If I change the first skript to

#!/usr/bin/perl use IO::Handle; #new STDOUT->autoflush; #new print "Please press return\n"; my $line = <STDIN>; print "OK, exit now\n";

the second script works as aspected:

TIME: Wed Jan 6 23:03:34 2010 OUTPUT: Please press return TIME: Wed Jan 6 23:03:34 2010 OUTPUT: exit

Now here is my question: Is there a way to prevent buffering without changing the called program? Why is $fh_out->autoflush(1) not working?

-Stefan

Replies are listed 'Best First'.
Re: IPC::Open3 buffering and autoflush
by chromatic (Archbishop) on Jan 06, 2010 at 22:24 UTC
    Why is $fh_out->autoflush(1) not working?

    Because the writing side isn't your filehandle.

    Is there a way to prevent buffering without changing the called program?

    If you set up a pseudo-tty, perhaps the OS would use line buffering on STDOUT instead. Use IO::Pty instead of IO::Handle. (Warning; I did not test this myself.)

Re: IPC::Open3 buffering and autoflush
by almut (Canon) on Jan 07, 2010 at 02:37 UTC

    As it seems you want to interact with an external console program, you might also want to look into Expect (which uses IO::Pty under the hood).

Re: IPC::Open3 buffering and autoflush
by urthwrm (Novice) on Jan 07, 2010 at 04:15 UTC
    I think the camel book answers your question quite nicely, so I'll simply just quote directly from it.

    The problem with standard I/O buffering is though your output filehandle is autoflushed so the process on the other end will see your data in a timely manner, you can't usually do anything to force it to return the favor. In some cases, for example "bc -l", it expects to operate over a pipe so therefor it flushes each output line, but it often seldom works this way unless you wrote the program yourself ( As you did with the script and turned off buffering on STDOUT). Various interactive programs also fail here, like FTP which wont do line buffering on a pipe, only on a TTY device.

    As mentioned, the IO::Pty and Expect modules provide a pseuedo-tty device, giving you line-buffering without needing to modify the program on the end of the pipe.

      Thanks for the suggestions. My main goal is to just log the output, grep different strings while the called program is running, highlight errors and other stuff. But in some situations I have to provide input. I thought that I could blame open3 because I thought that the called program whould inherit the stdout-filehandle and its properties, so I wrote a wrapper-script

      use IO::Handle; STDOUT->autoflush(1); exec ./program_to_launch

      And then I called the wrapperscript with open3. But it doesnt seem to work. I think I haven't really understand where the buffering actually takes place. I will now take a look at IO::Pty.

      -Stefan
Re: IPC::Open3 buffering and autoflush
by zentara (Cardinal) on Jan 07, 2010 at 12:21 UTC
    ....see IPC3 buffer limit problem for an intersting trick you can use to clear the pipe in tough applications

    also see How to use pty for both STDERR AND STDOUT in IPC::Run, while doing something different with each? and for a pty here is example code:

    #!/usr/bin/perl -w # Description: Fool a process into # thinking that STDOUT is a terminal, when in fact # it may be a file or a pipe. This can be useful # with programs like ps and w on linux... which # will trunc their output to the width of the # terminal, and, if they cannot detect the terminal # width, use a default 80 columns. Wouldn't it be # nice to say "ps -aux | grep etcshadow", and get # output that looks like when you just say "ps # -aux"? Well, that's the idea. #try ./pseudotty "xterm -e top" #or ./pseudotty top use warnings; use strict; use IO::Pty; die "usage: ptyexec command [args]\n" unless @ARGV; my $pty = IO::Pty->new; my $slave = $pty->slave; open TTY,"/dev/tty" or die "not connected to a terminal\n"; $pty->clone_winsize_from(\*TTY); close TTY; my $pid = fork(); die "bad fork: $!\n" unless defined $pid; if (!$pid) { # $slave->close(); open STDOUT,">&=".$pty->fileno() or die $!; exec @ARGV; }else{ $pty->close(); while (defined (my $line = <$slave>)) { print $line; } } #cleanup pty for next run $pty->close();

    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku
Re: IPC::Open3 buffering and autoflush
by puravida (Initiate) on Jul 03, 2014 at 23:19 UTC

    As many has pointed out IO::Pty is the right module to use. The buffer comes from the external program, not the IPC::Open3. Posted below is the complete program to demonstrate how this is done with IO::Pty.

    #!/usr/bin/env perl -w use strict; use IO::Pty; my $master = new IO::Pty; my $slave = $master->slave(); my $pid = fork(); die "Couldn't fork: $!" unless defined $pid; if ($pid) { $master->close_slave(); while ( !$master->eof ) { my $line = $master->getline; print "TIME: " . localtime() . " OUTPUT: " . $line; if ( $line =~ m/^Please press return/ ) { $master->print("abc\n"); } } wait(); exit $? >> 8; } else { $master->close(); $master->make_slave_controlling_terminal(); open( STDIN, ">&", $slave ) or die "Couldn't dup stdin: $!"; open( STDOUT, ">&", $slave ) or die "Couldn't dup stdout: $!"; open( STDERR, ">&", $slave ) or die "Couldn't dup stderr: $!"; # External program with buffered IO. system q( perl -e ' print "Please press return\n"; my $line = <STDIN>; print "OK, $line, exit now\n"; exit 14; ' ); }

    If you run the program, you will get this:

    TIME: Thu Jul  3 19:30:47 2014 OUTPUT: Please press return
    TIME: Thu Jul  3 19:30:47 2014 OUTPUT: abc
    TIME: Thu Jul  3 19:30:47 2014 OUTPUT: OK, abc
    TIME: Thu Jul  3 19:30:47 2014 OUTPUT: , exit now
    

    The second line comes from echo to the $master, how do I turn off echo?