Named pipes missing connections

bbfu has asked for the wisdom of the Perl Monks concerning the following question:

I'm having some problems with named pipes, and was hoping some monk could shed some light for me. The problem that I'm having is thus:

I have a program that sits waiting for some other program to open a named pipe, at which point it reads all the data from the named pipe, closes the pipe, and acts on the data (it's going to be a mail filter). It then goes back to the start, waiting for another program to open the pipe.

The problem is, if the writer opens the pipe too many times, in too short of a period, the reader ends up losing entire sessions of the opened pipe (ie, I never get partial data, just entire times that the pipe was opened by the writer are never seen by the reader).

When I try to reduce this to a simple example, however, I get a different problem. In the following simple programs, tstin.pl sometimes gets all the data from tstout.pl in a single connection, instead of one line per connection, as tstout.pl is actually sending it. So, closing the pipe in tstout.pl doesn't seem to propagate to tstin.pl for some reason.

It seems very strange to me, but perhaps I'm missing something. I look forward to any advice, or suggestions.

tstin.pl

#!/usr/bin/perl
use warnings;

use 5.6.0;

use IO::Handle;
STDOUT->autoflush(1);
STDERR->autoflush(1);

$SIG{PIPE} = sub { die "SIGPIPE: (@_) $!\n" };

$/ = undef;
while(1) {
  print "Opening pipe: ";
  open $pipe, "< tst.fifo" or die "Can't open pipe: $!\n";
  $pipe->autoflush(1);
  print "got pipe, reading:\n";
  print map ": $_\n", split "\n", scalar <$pipe>;
  print "Pipe done, closing: ";
  close $pipe or die "Can't close pipe: $!\n";
  print "done.\n";
  print "Taking a long time... ";
  sleep 3;
}

exit;
[download]

tstout.pl

#!/usr/bin/perl
use warnings;

use 5.6.0;

use IO::Handle;
STDOUT->autoflush(1);
STDERR->autoflush(1);

$SIG{PIPE} = sub { die "SIGPIPE: (@_) $!\n" };

$delay = shift || 0;

@data = map "$_\n", split "\n", <<'EOD';
This is some silly,
stuff to be sent
to the pipe where hopefully
it will be read.
EOD

while(@data and $_ = shift @data) {
  open $pipe, "> tst.fifo" or die "Can't open pipe: $!\n";
  $pipe->autoflush(1);
  print $pipe $_;
  close $pipe or die "Can't close pipe: $!\n";
  sleep $delay if $delay;
}

exit;
[download]

(Occasional) tstin.pl Output

Opening pipe: got pipe, reading:
: This is some silly,
: stuff to be sent
: to the pipe where hopefully
: it will be read.
Pipe done, closing: done.
Taking a long time... done.
[download]

bbfu
Seasons don't fear The Reaper.
Nor do the wind, the sun, and the rain.
We can be like they are.

Comment on Named pipes missing connections Select or Download Code

Replies are listed 'Best First'.
Re: Named pipes missing connections by abstracts (Hermit) on Dec 24, 2001 at 04:24 UTC
hello, It seems like your OS (Linux here) is buffering all successive open-write-close sequences together and lumping all data together that can be read with a single open-read-close sequence. Here is a trace of the system calls for both the writer (first) and the reader (second). `# this is the writer trace # this sequence is repeated 4 times open("tst.fifo", O_WRONLY\|O_CREAT\|O_TRUNC\|O_LARGEFILE, 0666) = 3 fstat64(3, {st_mode=S_IFIFO\|0644, st_size=0, ...}) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 fstat64(3, {st_mode=S_IFIFO\|0644, st_size=0, ...}) = 0 old_mmap(NULL, 4096, PROT_READ\|PROT_WRITE, MAP_PRIVATE\|MAP_ANONYMOUS, +-1, 0) = 0x40016000 write(3, "This is some silly,\n", 20) = 20 close(3) = 0 munmap(0x40016000, 4096) = 0` [download] # this is the reader trace # notice how we got all 82 bytes at one shot write(1, "Opening pipe: ", 14) = 14 open("tst.fifo", O_RDONLY\|O_LARGEFILE) = 3 fstat64(3, {st_mode=S_IFIFO\|0644, st_size=0, ...}) = 0 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 write(1, "got pipe, reading:\n", 19) = 19 fstat64(3, {st_mode=S_IFIFO\|0644, st_size=0, ...}) = 0 old_mmap(NULL, 4096, PROT_READ\|PROT_WRITE, MAP_PRIVATE\|MAP_ANONYMOUS, +-1, 0) = 0x40017000 read(3, "This is some silly,\nstuff to be "..., 4096) = 82 read(3, "", 4096) = 0 write(1, ": This is some silly,\n", 22) = 22 write(1, ": stuff to be sent\n", 19) = 19 write(1, ": to the pipe where hopefully\n", 30) = 30 write(1, ": it will be read.\n", 19) = 19 write(1, "Pipe done, closing: ", 20) = 20 close(3) = 0 munmap(0x40017000, 4096) = 0 write(1, "done.\n", 6) = 6 write(1, "Taking a long time... ", 22) = 22 time([1009160476]) = 1009160476 [download] So, is this a bug? Dunno, since I'm not familiar with the POSIX specs on pipes. Hope this helps. Aziz,,, Update: From the pipe(4) man page: `Under Linux, opening a FIFO for read and write will suc? ceed both in blocking and non-blocking mode. POSIX leaves this behaviour undefined. This can be used to open a FIFO for writing while there are no readers available. A pro? cess that uses both ends of the connection in order to communicate with itself should be very careful to avoid deadlocks.` [download] So, I guess under linux, the kernel choses to buffer the input when there are multiple succesive writes to a named pipe. Hope this helps...	[reply] [d/l] [select]
(bbfu) (in the OS) Re2: Named pipes missing connections by bbfu (Curate) on Dec 24, 2001 at 05:21 UTC
Alas, it seems you are correct and my problem is within the OS. Unfortunately, that means there's not really anything I can do about it, and must look for an alternative solution. Ah, well. C'est la vie. Thanks for your help (and termix, as well, for your suggestions)! <update>Thanks, premchai21. Never could speel... Especially not French. ;-)</update> bbfu Seasons don't fear The Reaper. Nor do the wind, the sun, and the rain. We can be like they are.	[reply]
Re: (bbfu) (in the OS) Re2: Named pipes missing connections by premchai21 (Curate) on Dec 24, 2001 at 08:45 UTC
Well, if your data is well structured enough, you could use some sort of delimited, either using $/ or just reading byte-by-byte and determining limits yourself. You then have to make sure that the delimiter never occurs in the data, though. Ah, well. Se la vi. I believe it's "C'est la vie" -- that's life. :-)	[reply]
Re: Named pipes missing connections by termix (Beadle) on Dec 24, 2001 at 04:43 UTC
Pipes don't serialize or check for things like two readers and writers. You can have a reader "A" open the pipe with writer "A", and then switch writers on the reader (by starting another writer "B" and THEN closing writer "A"). The reader sees an EOF only if a read attempt is made when no one is WRITING to it. This is further complicated by the fact that two different processes (Reader and writer) are not synched in other ways. I recommend: a lock file to make sure two writers don't touch the pipe at the same time. Make this lock on a separate lock file (no use locking a pipe you just opened if the idea is to not touch the pipe before locking). Serialize the communications (have each writer use a unique number). How do you get the unique number? Using the lock file from (1) could be a solution. I must admit I did not experiment with your code so there might be other reasons (but locking is a good idea if you have the remotest chance that two writers will need to talk at the same time). -- termix	[reply]
(bbfu) (further thoughts) Re: Named pipes missing connections by bbfu (Curate) on Dec 24, 2001 at 04:22 UTC
Sorry if the text above seems a little rushed. I was on my way out from work. :-) I double checked the main program, and it is actually suffering from the same issue as the reduced test. If I put a delay in the writer, everything works fine. I was looking, however, for something I could do in the reader, as that will be the only half I have any control over in the end. As far as I can tell, the problem seems to be that the reader is not seeing the closing of the pipe if another writer opens the pipe again soon enough. This seems to me to go against the point of the named pipe, though I suppose I could accept the rationalization that the reader is still "present" when the new writer opens the pipe (as the reader hasn't had time to close its end of the pipe yet), so it doesn't block, and there is still data comming in (from the new writer), so the reader doesn't see an eof... I guess. :-) I suppose I should also include the apropriate information about my perl and OS: `Summary of my perl5 (revision 5.0 version 6 subversion 1) configuratio +n: Platform: osname=linux, osvers=2.4.13, archname=i386-linux uname='linux duende 2.4.13 #1 wed oct 31 19:18:07 est 2001 i686 un +known '` [download] ... and so on. I really appreciate any info anyone can shed on this. TIA. bbfu Seasons don't fear The Reaper. Nor do the wind, the sun, and the rain. We can be like they are.	[reply] [d/l]