in reply to Buffered, bruised and broken

You are Suffering from Buffering. When you swap your output to file writes, the buffering behavior is swapped as well. You can repair this by making STDIO "hot" by setting the special variable $| to true. The following code behaves as you would expect.

## try.pl ##
use strict;
use warnings;
#use Inline C => Config => BUILD_NOISY => 1;

use Inline C => <<'END_C';

void foo(int i) {
        printf("From C: %d\n", i);
        /* Flush all IO buffers */
        fflush(NULL);
}


END_C

$| = 1;
for (my $i =0; $i < 10; ++$i) {
        print "From perl: $i\n";
        foo(++$i);
}

[download]

P.S. You didn't get bit this time, but you should always use strict;. Perhaps that was a transcription error, given that you use warnings;.

Replies are listed 'Best First'.
Re^2: Buffered, bruised and broken
by syphilis (Archbishop) on Jan 24, 2009 at 13:58 UTC
    You can repair this ... by setting ... $| to true

    I had tried that and found that it made no difference. But, you're right, it *does* fix the problem - and my original test of setting $| was apparently flawed. (I had tested it only on Windows, and there's a Windows bug that accounts for this.)

    But that solution, in itself, raises another question. Why should setting $| fix the problem ? Setting $| should flush only the perl buffer, shouldn't it ? And that buffer was already being flushed by the trailing "\n". I don't see why setting $| should make any difference ... yet it does.

    ... you should always use strict;. Perhaps that was a transcription error

    No, it wasn't a transcription error. If use of strictures has no bearing on a demo script, then I tend to dispense with it.

    Cheers,
    Rob
[reply]

      When STDOUT is connected to a terminal then it is "hot" by default and perl flushes its buffer when "\n" is written, but when it is not connected to a terminal, as when you redirect STDOUT to the file, then it is not "hot" by default and perl does not, by default, flush its buffer when "\n" is written. You can make it "hot" by setting "$|" to a true value (e.g. 1).

      You can use strace, on linux, to see when perl writes to the system. When I run your test program under strace I get the following:

      [ian@alula ~]$ strace perl ./test.pl >test.log

...

write(1, "From C: 1\n", 10)             = 10
write(1, "From C: 3\n", 10)             = 10
write(1, "From C: 5\n", 10)             = 10
write(1, "From C: 7\n", 10)             = 10
write(1, "From C: 9\n", 10)             = 10
write(1, "From perl: 0\nFrom perl: 2\nFrom p"..., 65) = 65
exit_group(0)                           = ?

      [download]

      Note that all the output from perl appears in a single write to the system. In this case, perl flushes its partially filled buffer as it prepares to exit.

      If I add "$| = 1;", then the strace output ends as follows.

      write(1, "From C: 1\n", 10)             = 10
write(1, "From perl: 2\n", 13)          = 13
write(1, "From C: 3\n", 10)             = 10
write(1, "From perl: 4\n", 13)          = 13
write(1, "From C: 5\n", 10)             = 10
write(1, "From perl: 6\n", 13)          = 13
write(1, "From C: 7\n", 10)             = 10
write(1, "From perl: 8\n", 13)          = 13
write(1, "From C: 9\n", 10)             = 10
exit_group(0)                           = ?

      [download]

      In this case, because STDOUT was made "hot" by setting $| to 1, perl flushed its buffer to the system every time "\n" was written.

[reply]
[d/l]
[select]

        In this case, because STDOUT was made "hot" by setting $| to 1, perl flushed its buffer to the system every time "\n" was written.

        That's not quite true. By setting $| to 1, Perl flushes the buffer every time anything is written. There are three states, not two: 

        $ strace perl -e'$|=0; print for "a", "b\n", "c\n"' >/dev/tty
write(1, "ab\n", 3)                     = 3
write(1, "c\n", 2)                      = 2

$ strace perl -e'$|=0; print for "a", "b\n", "c\n"' >/dev/null
write(1, "ab\nc\n", 5)                  = 5

$ strace perl -e'$|=1; print for "a", "b\n", "c\n"'
write(1, "a", 1)                        = 1
write(1, "b\n", 2)                      = 2
write(1, "c\n", 2)                      = 2

        [download]
[reply]
[d/l]
[select]
        ... when you redirect STDOUT to the file, then ... perl does not, by default, flush its buffer when "\n" is written.

        Aaah, I see. Thanks ig, kennethk.

        Cheers,
        Rob
[reply]

      But that solution, in itself, raises another question. Why should setting $| fix the problem ?

      Output buffers are not provided by the OS. Each library that does output buffering provides its own system. When you are using the C lib to do output, you use its buffers. PerlIO apparently does not use the C lib for output and thus has independent buffers.

      The real solution is to use the functions documented in perlapio instead of the C output functions. ( Oops, you already said that )

      Setting $| should flush only the perl buffer, shouldn't it ? And that buffer was already being flushed by the trailing "\n"

      "\n" only flushes STDOUT when it's connected to a terminal. By redirecting the output, you turn on full buffering. $|=1; turns it off, returning to a situation similar to when you didn't redirect STDOUT (but different since it's not even line buffered now).

      STDOUT->autoflush(0)STDOUT is connected to a terminalSTDOUT is line buffered (flushed by "\n")
      STDOUT->autoflush(0)STDOUT isn't connected to a terminalSTDOUT is block buffered (not flushed by "\n")
      STDOUT->autoflush(1)STDOUT isn't buffered (flushed after every print)
       
      STDERR->autoflush(0)STDERR isn't buffered (flushed after every print)
      STDERR->autoflush(1)STDERR isn't buffered (flushed after every print)
       
      $fh->autoflush(0)$fh is block buffered (not flushed by "\n")
      $fh->autoflush(1)$fh isn't buffered (flushed after every print)

      (Where $fh is any handle other than STDOUT and STDERR)

      Update: Apparently, one cannot turn off autoflush on STDERR, so I updated the table.

[reply]
[d/l]
[select]
      I don't see why setting $| should make any difference ... yet it does.

      I was surprised this worked, too. The parallel case in the linked documentation was for a system command w/ output (search: ls), so I chalk it up to the Perl buffer and c buffer actually being implemented separately.

      If use of strictures has no bearing on a demo script, then I tend to dispense with it.

      At the least, it tells folks who are looking at the scripts on-line not to worry about red herrings...

      Glad my fix helped, at least with the Unix side.

[reply]

In Section Seekers of Perl Wisdom