syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I thought I understood how to deal with the mixing of XS and perl buffers ... but apparently not. Consider this simple Inline::C demo script: 
## try.pl ##
use warnings;
use Inline C => Config =>
    BUILD_NOISY => 1;

use Inline C => <<'END_C';

void foo(int i) {
        printf("From C: %d\n", i);
        /* Flush all IO buffers */
        fflush(NULL);
}


END_C

for ($i =0; $i < 10; ++$i) {
        print "From perl: $i\n";
        foo(++$i);
}

[download]
There's no problem with that - it outputs (as I expect): 
From perl: 0
From C: 1
From perl: 2
From C: 3
From perl: 4
From C: 5
From perl: 6
From C: 7
From perl: 8
From C: 9

[download]
But when (with perl-5.10.0, both Linux and Windows) I run the script as perl try.pl >out.txt I find that out.txt contains: 
From C: 1
From C: 3
From C: 5
From C: 7
From C: 9
From perl: 0
From perl: 2
From perl: 4
From perl: 6
From perl: 8

[download]
I don't understand why the output has changed - I expected the flushing of the buffers to take care of things for me ... but it obviously didn't. Can anyone enlighten me ?

This is pretty much an academic question - I know of a number of ways to modify that Inline::C script so that the output, when re-directed to out.txt, is as expected/desired. (The simplest way is to replace "printf" with "PerlIO_stdoutf".) But my question is "Why does the above script fail to re-direct correctly ?"

Update: Hmmmm ... when I said "I know of a number of ways ...", the "number" I was thinking of was, in fact, "1" :-)
There are other ways of getting the correct output into out.txt, but when it comes to command line re-direction, the *only* way I know of is to replace "printf" with "PerlIO_stdoutf".

Cheers,
Rob

Replies are listed 'Best First'.
Re: Buffered, bruised and broken
by kennethk (Abbot) on Jan 24, 2009 at 12:39 UTC

    You are Suffering from Buffering. When you swap your output to file writes, the buffering behavior is swapped as well. You can repair this by making STDIO "hot" by setting the special variable $| to true. The following code behaves as you would expect.

    ## try.pl ##
use strict;
use warnings;
#use Inline C => Config => BUILD_NOISY => 1;

use Inline C => <<'END_C';

void foo(int i) {
        printf("From C: %d\n", i);
        /* Flush all IO buffers */
        fflush(NULL);
}


END_C

$| = 1;
for (my $i =0; $i < 10; ++$i) {
        print "From perl: $i\n";
        foo(++$i);
}

    [download]

    P.S. You didn't get bit this time, but you should always use strict;. Perhaps that was a transcription error, given that you use warnings;.

[reply]
[d/l]
[select]
      You can repair this ... by setting ... $| to true

      I had tried that and found that it made no difference. But, you're right, it *does* fix the problem - and my original test of setting $| was apparently flawed. (I had tested it only on Windows, and there's a Windows bug that accounts for this.)

      But that solution, in itself, raises another question. Why should setting $| fix the problem ? Setting $| should flush only the perl buffer, shouldn't it ? And that buffer was already being flushed by the trailing "\n". I don't see why setting $| should make any difference ... yet it does.

      ... you should always use strict;. Perhaps that was a transcription error

      No, it wasn't a transcription error. If use of strictures has no bearing on a demo script, then I tend to dispense with it.

      Cheers,
      Rob
[reply]

        When STDOUT is connected to a terminal then it is "hot" by default and perl flushes its buffer when "\n" is written, but when it is not connected to a terminal, as when you redirect STDOUT to the file, then it is not "hot" by default and perl does not, by default, flush its buffer when "\n" is written. You can make it "hot" by setting "$|" to a true value (e.g. 1).

        You can use strace, on linux, to see when perl writes to the system. When I run your test program under strace I get the following:

        [ian@alula ~]$ strace perl ./test.pl >test.log

...

write(1, "From C: 1\n", 10)             = 10
write(1, "From C: 3\n", 10)             = 10
write(1, "From C: 5\n", 10)             = 10
write(1, "From C: 7\n", 10)             = 10
write(1, "From C: 9\n", 10)             = 10
write(1, "From perl: 0\nFrom perl: 2\nFrom p"..., 65) = 65
exit_group(0)                           = ?

        [download]

        Note that all the output from perl appears in a single write to the system. In this case, perl flushes its partially filled buffer as it prepares to exit.

        If I add "$| = 1;", then the strace output ends as follows.

        write(1, "From C: 1\n", 10)             = 10
write(1, "From perl: 2\n", 13)          = 13
write(1, "From C: 3\n", 10)             = 10
write(1, "From perl: 4\n", 13)          = 13
write(1, "From C: 5\n", 10)             = 10
write(1, "From perl: 6\n", 13)          = 13
write(1, "From C: 7\n", 10)             = 10
write(1, "From perl: 8\n", 13)          = 13
write(1, "From C: 9\n", 10)             = 10
exit_group(0)                           = ?

        [download]

        In this case, because STDOUT was made "hot" by setting $| to 1, perl flushed its buffer to the system every time "\n" was written.

[reply]
[d/l]
[select]

        But that solution, in itself, raises another question. Why should setting $| fix the problem ?

        Output buffers are not provided by the OS. Each library that does output buffering provides its own system. When you are using the C lib to do output, you use its buffers. PerlIO apparently does not use the C lib for output and thus has independent buffers.

        The real solution is to use the functions documented in perlapio instead of the C output functions. ( Oops, you already said that )

        Setting $| should flush only the perl buffer, shouldn't it ? And that buffer was already being flushed by the trailing "\n"

        "\n" only flushes STDOUT when it's connected to a terminal. By redirecting the output, you turn on full buffering. $|=1; turns it off, returning to a situation similar to when you didn't redirect STDOUT (but different since it's not even line buffered now).

        STDOUT->autoflush(0)STDOUT is connected to a terminalSTDOUT is line buffered (flushed by "\n")
        STDOUT->autoflush(0)STDOUT isn't connected to a terminalSTDOUT is block buffered (not flushed by "\n")
        STDOUT->autoflush(1)STDOUT isn't buffered (flushed after every print)
         
        STDERR->autoflush(0)STDERR isn't buffered (flushed after every print)
        STDERR->autoflush(1)STDERR isn't buffered (flushed after every print)
         
        $fh->autoflush(0)$fh is block buffered (not flushed by "\n")
        $fh->autoflush(1)$fh isn't buffered (flushed after every print)

        (Where $fh is any handle other than STDOUT and STDERR)

        Update: Apparently, one cannot turn off autoflush on STDERR, so I updated the table.

[reply]
[d/l]
[select]
        I don't see why setting $| should make any difference ... yet it does.

        I was surprised this worked, too. The parallel case in the linked documentation was for a system command w/ output (search: ls), so I chalk it up to the Perl buffer and c buffer actually being implemented separately.

        If use of strictures has no bearing on a demo script, then I tend to dispense with it.

        At the least, it tells folks who are looking at the scripts on-line not to worry about red herrings...

        Glad my fix helped, at least with the Unix side.

[reply]

Back to Seekers of Perl Wisdom