thecap has asked for the wisdom of the Perl Monks concerning the following question:

I am using Open2 to run a C program that outputs a line with a timestamp in reply to each input line. The problem is that the reply lines are buffered together.

Desired results (happens when program is run from terminal or if C program calls fflush(stdout) after each printf): 

Input1
time(): Output1
END
time(): exit

[download]

Observed with Open2 and no fflush in the C program: 

Input1
END
time(): Output1
time(): exit

[download]

I believe the problem is that C program's libc tries to detect if stdout is connected to a terminal. If it is connected to a terminal it uses line buffering, otherwise it uses full buffering. How can I make the stdout of a program started with Open2 appear to be connected to a terminal?

I do not want to change the source of the C program. I am using IO::Select to see if there is any output as the actual program I want to use will only sometimes produce output.

I tried looking at the glibc-2.3.2 source to find how it decides what buffering to use on stdout, but could not work out how the line "./sysdeps/standalone/open.c: setvbuf( stdout, NULL, _IOLBF, BUFSIZ );" is called.

Here is a strace dump of running the C program with Open2: 

read(0, "Input1\n", 4096)               = 7
time(NULL)                              = 1066404185
read(0, "END\n", 4096)                  = 4
time(NULL)                              = 1066404190
write(1, "1066404185: \"Input1\"\n1066404190:"..., 38) = 38

[download]

Here is a run with the same inputs from the shell: 

read(0, "Input1\n", 1024)               = 7
time(NULL)                              = 1066404693
write(1, "1066404693: \"Input1\"\n", 21) = 21
read(0, "END\n", 1024)                  = 4
time(NULL)                              = 1066404697
write(1, "1066404697: exit\n", 17)      = 17

[download]

Thank you for any ideas you may have to help me,
Tom

Replies are listed 'Best First'.
Re: How to change Open2 input buffering (fix C RTL!)
by tye (Sage) on Oct 17, 2003 at 16:05 UTC

    This has been a known problem for well over a decade. There are several ways to fix this that would be trivial to implement. Probably the best place to start is to get GNU to add such a feature to their C RTL so that others might follow suit.

    If you have access to your C RTL source code and can build a new shared library, then you can even fix this yourself (even for C programs that you don't have the source code to -- unless they were statically linked).

    The simplest fix I'd use would be an environment variable that, when set, tells the C RTL to act like isatty(2) returned true for stdout (etc.) and so would use "line buffering" instead of "full buffering" (for example, export LINEBUFFEREDPIPES=1).

    The current "solution" is to use pseudo ttys instead of pipes. This is quite the aircraft carrier to swat a fly, but that gives you another route to investigate.

                    - tye
[reply]
[d/l]
Re: How to change Open2 input buffering
by ptkdb (Monk) on Oct 17, 2003 at 16:34 UTC
    Another solution in the brute force category, is there any input that you can give to the C program that will get it to produce output, thereby pushing data out of the buffer?

    You may just have to bite the bullet and change the program.

    At a guess, you're trying to solve some production issue problem with perl script, but the the C program in question belongs to some other group, is legacy code, or is under "Ivory Tower Source Control", or some other situation that makes modifying it, even with just 1 line of code, difficult.

    I know that in certain organizations modifying a legacy program is tougher than passing a bill through Congress, but if they have any kind of tracking system, such as ClearQuest, file the 'Request for an Enhancement', or even elevate it to the level of a bug on the basis of "failure to flush output, prevents programs use with automated scripting tools."

[reply]
Re: How to change Open2 input buffering
by sgifford (Prior) on Oct 17, 2003 at 20:14 UTC
    You need to use a pseudo-tty to solve this problem. IO::Pty is a fairly easy way to solve this. Here's an example: 
    #!/usr/bin/perl -w

use strict;

use IO::Pty;

my $pty = IO::Pty->new
    or die "Pty error: $!\n";
my $slave = $pty->slave
    or die "Pty slave error: $!\n";

# Run cat with stdout on the PTY.
pipe(PIPEREAD,PIPEWRITE);
if (fork()==0)
{
  # Child
  close(PIPEWRITE);
  close($slave);
  close(STDOUT)
      or die "Couldn't close STDOUT: $!\n";
  open(STDOUT,">&".$pty->fileno)
      or die "Couldn't re-open STDOUT: $!\n";
  close(STDIN)
      or die "Couldn't close STDIN: $!\n";
  open(STDIN,"<&PIPEREAD")
      or die "Couldn't re-open STDIN: $!\n";
  exec("/bin/cat")
      or die "Couldn't exec /bin/cat: $!\n";
}

# Parent
close(PIPEREAD);
close($pty);

select(PIPEWRITE);
$|=1;
select(STDOUT);

warn "Parent running\n";

foreach my $val (1..10) {
  print PIPEWRITE "$val\n";
  $_ = <$slave>;
  print "$_";
}

    [download]
[reply]
[d/l]
[select]
      Thank you tye, ptkdb, and sgifford for your excellent replies.

      The pseudo-tty code sgifford posted worked wonderfully in Linux, but not when I moved to my final target, SunOS. I eventually found a small C program called ircflush, part of the ircII distribution that does the same thing and seems to work in SunOS. I also looked into using IPC::Run but it does not connect file handles to pseudo-ttys.

[reply]
        Huh. What version of SunOS?
[reply]

Back to Seekers of Perl Wisdom