Re^2: Using pipe and reading from the stdin (buffering)

I've run into that too, but I've never had to do too much to work around it...

$| = 1;
[download]

or more readably

use English '-no_match_vars';
$OUTPUT_AUTOFLUSH = 1;
[download]

will solve the problem for a pipe that has only Perl scripts in it. It needs to go on the stdout of each script in the pipe. (My experience so far has always been that the stdin side of a pipe will read the next line when the line feed arrive, but you may want to check that with your operating system...and your Perl version, and your C libraries...)

Some other common utilities will work in pipelines, but some insist on buffering their output. Which utilities are which? It's probably simpler to find out by experiment than to find the documentation, if it exists.

If you find that grep or cut on your system are holding their output in buffers, just rewrite them as perl scripts.

Comment on Re^2: Using pipe and reading from the stdin (buffering) Select or Download Code

Replies are listed 'Best First'.
Re^3: Using pipe and reading from the stdin (buffering) by tye (Sage) on Oct 27, 2006 at 15:29 UTC
Of course, `$\|= 1;` won't do the slightest good here. There is nothing Perl can do to unbuffer the output of the `tail` command that it is reading from (just to be clear). We are lucky that Perl often makes "writing your own replacement" relatively easy. You are lucky that you've "never had to do too much to work around it". Rewriting every command in the pipeline in Perl can be a lot of work or may not be possible. Most of the time we are lucky in that the output flows fairly continuously until EOF and so the buffering at most introduces a slight delay (and, in theory, makes the process go faster, though I'm suspicious of such pre-optimization thinking and suspect you'd be hard pressed to notice an efficiency difference between line buffering and full buffering). It is still rather silly and sad that such a simple feature that would be trivial to add a tiny bit of external control to and has caused so much grief over the previous decades hasn't been addressed. If I already had my fingers in GNU source code, I'd certainly submit a patch to the trivial: `setvbuf( STDOUT, NULL, isatty(1) ? _IOLBF : _IOFBF, BUFSIZE );` [download] To do something more like: `{ char* bufPref= getenv( "CRTL_STDOUT_BUF_MODE" ); int bufMode= NULL == bufPref ? ( isatty(1) ? _IOLBF : _IOFBF ) : "L"==bufPref ? _IOLBF : "F"==bufPref ? _IOFBF : _I +ONBF; } setvbuf( STDOUT, NULL, bufMode, BUFSIZE ); }` [download] - tye	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: Using pipe and reading from the stdin (buffering)
by tye (Sage) on Oct 27, 2006 at 15:29 UTC

Of course, $|= 1; won't do the slightest good here. There is nothing Perl can do to unbuffer the output of the tail command that it is reading from (just to be clear).

We are lucky that Perl often makes "writing your own replacement" relatively easy. You are lucky that you've "never had to do too much to work around it". Rewriting every command in the pipeline in Perl can be a lot of work or may not be possible.

Most of the time we are lucky in that the output flows fairly continuously until EOF and so the buffering at most introduces a slight delay (and, in theory, makes the process go faster, though I'm suspicious of such pre-optimization thinking and suspect you'd be hard pressed to notice an efficiency difference between line buffering and full buffering).

It is still rather silly and sad that such a simple feature that would be trivial to add a tiny bit of external control to and has caused so much grief over the previous decades hasn't been addressed. If I already had my fingers in GNU source code, I'd certainly submit a patch to the trivial:

    setvbuf( STDOUT, NULL, isatty(1) ? _IOLBF : _IOFBF, BUFSIZE );
[download]

To do something more like:

    {
        char* bufPref= getenv( "CRTL_STDOUT_BUF_MODE" );
        int   bufMode=
            NULL == bufPref
                ? ( isatty(1) ? _IOLBF : _IOFBF )
                : "L"==*bufPref ? _IOLBF : "F"==*bufPref ? _IOFBF : _I
+ONBF;
        }
        setvbuf( STDOUT, NULL, bufMode, BUFSIZE );
    }
[download]

- tye

[reply]
[d/l]
[select]