buffering when reading from unnamed pipe with child process

Santasha has asked for the wisdom of the Perl Monks concerning the following question:

This code...

#!/usr/bin/perl

use strict;
use warnings;

use IO::Pipe;
use IO::Handle;
use IO::Select;

my $pipe = IO::Pipe->new();

my $pid = fork();
die "failed to fork: $!" unless defined $pid;

if ($pid) {
    $pipe->reader();

    my $select = IO::Select->new();
    $select->add($pipe);

    while (1) {
        my @ready = $select->can_read(1);

        foreach my $h (@ready) {
            my $line = <$h>;
            chomp $line;
            print "PARENT sees <$line>\n";
        }
    }
}
else {
    $pipe->writer();
    $pipe->autoflush(1);

    my $c = 0;
    while (1) {
        $c++;
        print "CHILD writing <$c>\n";
        $pipe->print("$c\n");
        sleep 1 unless $c % 5; # pause after each fifth line
    }
}
[download]

produces output like this...

CHILD writing <1>
CHILD writing <2>
CHILD writing <3>
CHILD writing <4>
CHILD writing <5>
PARENT sees <1>
CHILD writing <6>
CHILD writing <7>
CHILD writing <8>
CHILD writing <9>
CHILD writing <10>
PARENT sees <2>
PARENT sees <3>
PARENT sees <4>
PARENT sees <5>
PARENT sees <6>
CHILD writing <11>
CHILD writing <12>
CHILD writing <13>
CHILD writing <14>
CHILD writing <15>
PARENT sees <7>
PARENT sees <8>
PARENT sees <9>
PARENT sees <10>
PARENT sees <11>
[download]

And so on.

What do I need to do to avoid the buffering of child writes that seems to be occuring?

This is on Debian/Linux.

Comment on buffering when reading from unnamed pipe with child process Select or Download Code

Replies are listed 'Best First'.
Re: buffering when reading from unnamed pipe with child process by tybalt89 (Monsignor) on Nov 19, 2020 at 23:05 UTC
#!/usr/bin/perl use strict; # https://perlmonks.org/?node_id=11123851 use warnings; use IO::Pipe; use IO::Handle; use IO::Select; my $pipe = IO::Pipe->new(); my $pid = fork(); die "failed to fork: $!" unless defined $pid; if ($pid) { $pipe->reader(); my $select = IO::Select->new(); $select->add($pipe); my $line = ''; while (1) { my @ready = $select->can_read(1); foreach my $h (@ready) { # my $line = <$h>; if( sysread $h, $line, 8192, length $line ) { # chomp $line; while( $line =~ s/(.*)\n// ) { print "PARENT sees <$1>\n"; } } else { close $pipe }; } } } else { $pipe->writer(); $pipe->autoflush(1); my $c = 0; while (1) { $c++; print "CHILD writing <$c>\n"; $pipe->print("$c\n") or die; sleep 1 unless $c % 5; # pause after each fifth line } } [download] CHILD writing <1> CHILD writing <2> CHILD writing <3> CHILD writing <4> CHILD writing <5> PARENT sees <1> PARENT sees <2> PARENT sees <3> PARENT sees <4> PARENT sees <5> CHILD writing <6> CHILD writing <7> CHILD writing <8> CHILD writing <9> CHILD writing <10> PARENT sees <6> PARENT sees <7> PARENT sees <8> PARENT sees <9> PARENT sees <10> CHILD writing <11> CHILD writing <12> CHILD writing <13> CHILD writing <14> CHILD writing <15> PARENT sees <11> PARENT sees <12> PARENT sees <13> PARENT sees <14> PARENT sees <15> [download]	[reply] [d/l] [select]
Re^2: buffering when reading from unnamed pipe with child process by Santasha (Novice) on Nov 20, 2020 at 08:16 UTC
Greatly appreciated, thank you.	[reply]
Re^3: buffering when reading from unnamed pipe with child process by perlfan (Parson) on Nov 20, 2020 at 19:42 UTC
I have found setting autoflush in the parent via `$\|++;` also helps; but using the pipe interface directly it also makes sense to do it this way. My recommendation in addition to the explicit flush is to set autoflush in the parent.	[reply] [d/l]
Re: buffering when reading from unnamed pipe with child process by GrandFather (Saint) on Nov 19, 2020 at 23:29 UTC
After the fork you have two processes which each get slices of time in which to do some work. That means that the nature of that work is bursty - busy for a while, idle for a while - depending on what else is going on in the machine so for both producer and consumer the processing is in bursts and that is essentially out of the control of the processes. It's not the pipe that is introducing the burstiness through buffering, but it is a fundamental property of preemptive multi tasking operating systems. You could introduce semaphores or mutexes to lock step the two processes, but then you probably lose the advantage of having two processes anyway. Update: that might make sense if the OP didn't have a sleep generating output in bursts. See tybalt89's reply below. Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond	[reply]
Re^2: buffering when reading from unnamed pipe with child process by tybalt89 (Monsignor) on Nov 20, 2020 at 00:38 UTC
Nope. The first buffered read `<$h>` grabs all five of the first sent lines, and leaves the handle empty, thus the delay for the second read. It's not until the 6th line comes in that the program can get to the second read. The usual advice is to not mix buffered and un-buffered I/O, and this code is the perfect example of why. IO::Select operates on unbuffered data while readline is buffered. That's why I used sysread (which is unbuffered) rather than readline. NOTE: Using sysread may get you more than one line at a time (or even partial lines) which is why I added the loop that extracts and deletes one while line at a time.	[reply] [d/l]