Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Malfunctioning select() call on a FIFO

by Llew_Llaw_Gyffes (Scribe)
on Jan 06, 2010 at 17:22 UTC ( [id://815950]=perlquestion: print w/replies, xml ) Need Help??

Llew_Llaw_Gyffes has asked for the wisdom of the Perl Monks concerning the following question:

Brethren,

I have a mystery to present today.

I have a threaded Perl ICB client. (If you don't know what ICB is, don't worry about it, that's not important to the question.) It was written as a multithreaded application for several reasons, not the least of them being to learn how to use Perl threads. One of its six total threads creates a FIFO, then listens on it for external input piped into the client and passes anything it receives to the input thread, in order to allow external files or program output to be piped into the program as though typed or pasted in directly. The handler for that FIFO is the following function:

sub fifocmd { my ($output_window, $input_window, $status_line, $connection) = @_ +; my ($fifodata, $rin, $rout); if (mkfifo($icbmfifo, oct('0600'))) { icb_debug ($output_window, 1, "fifocmd() opening command FIFO +%s", $icbmfifo); # We have to use sysopen() here rather than 'open (FIFO, $icbm +fifo);' # because on many preinstalled Perls, open() does NOT use O_NO +NBLOCK. # This causes breakage, as in the open() call never returns un +til and # unless something writes to the other end of the pipe. sysopen (FIFO, $icbmfifo, O_RDONLY|O_NONBLOCK); icb_debug ($output_window, 1, "Command FIFO %s is open", $icbm +fifo); while ($input_thread_running > 0) { $rout = $rin = $fifodata = ''; vec($rin, fileno(FIFO), 1) = 1; select($rout = $rin, undef, undef, 0.01); if (vec($rout, fileno(FIFO), 1)) { chomp($fifodata = <FIFO>); icb_debug ($output_window, 1, "Read '%s' from FIFO\n", + $fifodata); next if ($fifodata eq '/x' || $fifodata eq '/exit' || +$fifodata eq '/quit'); handle_input($fifodata, $output_window, $input_window, $status_line, $connection) if (length($fifodata)); } threads->yield if ($input_thread_running > 0); } icb_debug ($output_window, 1, "Input thread stopped; fifocmd() + exiting"); close (FIFO); unlink ($icbmfifo); } else { icb_print ($output_window, 'status', "Could not open %s as FIF +O", $icbmfifo); icb_print ($output_window, 'status', "Remote control functiona +lity will not be available."); } }

The last time I used the external input FIFO functinality, which was quite some time ago, it worked perfectly. I had cause to use it again this morning, and found that it has begun malfunctioning. As soon as anything is written to the FIFO, the FIFO input is processed as it should be, but then CPU utilization goes to 100% and stays there. The debug asserts that I inserted into the code show that after the initial handling of the input, the function goes into a tight loop repeatedly reading nothing from the FIFO:

Group: *test* (mil) Mod: Testric Topic: (None) *Testric - 11:43:09 alaric@babcom.com <[DEBUG]> fifocmd() opening command FIFO /home/alaric/.icbm/SOCKETS/ +ICBM-cmdfi fo.22166 <[DEBUG]> Command FIFO /home/alaric/.icbm/SOCKETS/ICBM-cmdfifo.22166 + is open <[DEBUG]> Read '/.' from FIFO Group: *test* (mil) Mod: Testric Topic: (None) *Testric - 11:43:09 alaric@babcom.com <[DEBUG]> Read '' from FIFO <[DEBUG]> Read '' from FIFO <[DEBUG]> Read '' from FIFO <[DEBUG]> Read '' from FIFO <[DEBUG]> Read '' from FIFO

It appears that once the FIFO has been written to, the select() ALWAYS says there is unread data waiting on the FIFO, yet nothing can be actually read from it. During the tight loop, an eof(FIFO) call inserted at the beginning of the innermost loop always returns true. There is no data on the FIFO to read; but the select() call thinks there is. I see that 'perldoc perlfunc' says:

Note: on some Unixes, the select(2) system call may report a socket file descriptor as "ready for reading", when actually no data is available, thus a subsequent read blocks. It can be avoided using always the O_NONBLOCK flag on the socket. See select(2) and fcntl(2) for further details.

However, I am already using O_NONBLOCK. It also says:

WARNING: One should not attempt to mix buffered I/O (like "read" or <FH>) with "select", except as permitted by POSIX, and even then only on POSIX systems. You have to use "sysread" instead.

But I have tried replacing the line:

chomp($fifodata = <FIFO>);

with the following construct:

my $l; while (sysread(FIFO, $l, 255)) { $fifodata .= $l; }

I have also tried replacing:

select($rout = $rin, undef, undef, 0.01); if (vec($rout, fileno(FIFO), 1)) {

with:

if (select($rout = $rin, undef, undef, 0.01)) {

or even:

if (select($rout = $rin, undef, undef, undef)) {

The behavior remains unchanged, though. No matter what I do, the select() continues to ALWAYS immediately show the FIFO as having data pending after it has been written to once.

Can anyone enlighten me as to why this should be, and what I can do to fix it?

Replies are listed 'Best First'.
Re: Malfunctioning select() call on a FIFO
by ikegami (Patriarch) on Jan 06, 2010 at 17:52 UTC

    You're using sysread, right? When sysread return zero bytes read, it indicates EOF. For a fifo, that means the other end of the fifo was closed. The fifo is still closed every subsequent time you call select, so select keeps signaling that the handle needs to be serviced. Similarly, if sysread returns undef, it indicates an error occurred.

    Update: Disambiguated pronouns and clarified what EOF means for a fifo.

      I tried switching to a sysread(FIFO, ...) from using <FIFO> in the hopes that it might solve the problem, yes. It didn't make any difference, though. What's puzzling me is that this code USED to work, and now it doesn't, but the code hasn't been changed between when it worked and now.

Re: Malfunctioning select() call on a FIFO
by ikegami (Patriarch) on Jan 06, 2010 at 18:09 UTC

    One should not attempt to mix buffered I/O (like "read" or <FH>) with "select"

    Most definitely. Data could be waiting in the buffer and select wouldn't know anything about it.

    while (sysread(FIFO, $l, 255)) { $fifodata .= $l; }

    Using while there defies the purpose of using select. If there's less than 256 bytes available, sysread will block the second time through the loop. You want:

    my $rv = sysread(FIFO, $fifodata, 64*1024, length($fifodata));

    (Big is good. It's not a problem if fewer bytes are available.)

    The "ready for reading" half of your select loop should look something like:

    my $rv = sysread(FIFO, $fifodata, 64*1024, length($fifodata)); if (!defined($rv)) { ... handle error ... } if (!$rv) { ... handle eof. There might be a partial message in $fifodata ... } while (...$fifodata contains a complete message...) { ...extract message from $fifodata... ...process message... }

    For fixed length records, the bottom loop might look like

    while (length($fifodata) > $MSG_SIZE) { my $msg = substr($fifodata, 0, $MSG_SIZE, ''); process_msg($msg); }

    For variable length records, s/// is convenient

    while ($fifodata =~ s/^([^\n]*\n)//) { my $msg = $1; process_msg($msg); }

      The correction on the usage of sysread() is taken. However, I have to point out that this misses the point of the problem. My problem is not how to correctly read from the FIFO; a simple "$fifodata = <FIFO>;" has always worked here (though I'll admit sysread() is probably better). The problem is how to make the select() on the FIFO work properly again instead of always returning true, so that the handler sleeps until there's data available on the FIFO instead of going into a tight loop and repeatedly trying to read from the FIFO as fast as the CPU can execute the inner loop. The only reason I even tried using sysread() instead was in case <FIFO> was somehow leaving something on the FIFO that was making select() think new data had become available to read.

      I point out again that this select() call used to work, and has not been changed since then; it's just ... stopped working. The "if (select(...)) {...}" call, on the FIFO, has become effectively an "if (1) {...}" after the first time the other end of the FIFO is written to.

      Just to make sure the other end was being properly closed, I wrote this little test tool:

      #!/usr/bin/perl my $fifo = shift(@ARGV) || die "No FIFO"; if (-p $fifo) { open(FIFO, ">$fifo") || die "Cannot open $fifo for writing"; print FIFO "/.\n" || die "Cannot write to $fifo"; close(FIFO) || die "Could not close $fifo"; } else { print "$p is not a FIFO.\n"; }

      I start the client, it runs normally with negligible CPU utilization; I run the test script; it opens the FIFO, writes to it, closes the FIFO, and exits; the client executes the command sent through the FIFO; and client CPU utilization goes immediately to 100% and stays there, because every select() call on the FIFO after the first command is read from it is saying that there is data on the FIFO to read, even after the writeable end of the FIFO has been closed.

      Thinking about it, it's been long enough since I last used this functionality of the client that it's entirely possible that it stopped working when I upgraded from Perl-5.8 to Perl-5.10, or possibly even when I upgraded from Perl-5.6 to Perl-5.8.

        a simple "$fifodata = <FIFO>;" has always worked here (though I'll admit sysread() is probably better).

        Is is better simply because using buffered IO with select is wrong.

        The only reason I even tried using sysread() instead was in case <FIFO> was somehow leaving something on the FIFO

        Leaving stuff on the FIFO is fine. <FIFO> is bad because it can return less than what it reads from the FIFO. That causes select to block when it shouldn't. For example,

        use IO::Select qw( ); use IO::Handle qw( ); pipe(my $rfh, my $wfh) or die; $wfh->autoflush(1); if (fork()) { my $sel = IO::Select->new($rfh); while ($sel->can_read()) { my $got = <$rfh>; last if !defined($got); chomp $got; print("It took ", (time()-$got), " seconds to get the msg\n"); } } else { for (;;) { print($wfh time(), "\n") or die; print($wfh time(), "\n") or die; sleep(3); } }
        It took 0 seconds to get the msg It took 3 seconds to get the msg It took 0 seconds to get the msg It took 3 seconds to get the msg It took 0 seconds to get the msg ...

        But with sysread:

        my $sel = IO::Select->new($rfh); my $buf = ''; while ($sel->can_read()) { sysread($rfh, $buf, 64*1024, length($buf)) or last; while ($buf =~ s/^(.*)\n//) { my $got = $1; print("It took ", (time()-$got), " seconds to get the msg\ +n"); } }
        It took 0 seconds to get the msg It took 0 seconds to get the msg It took 0 seconds to get the msg It took 0 seconds to get the msg It took 0 seconds to get the msg It took 0 seconds to get the msg ...

        I point out again that this select() call used to work, and has not been changed since then;

        And select is still working properly. The problem has nothing to do with select.

        because every select() call on the FIFO after the first command is read from it is saying that there is data on the FIFO to read

        No, it's saying that the handle needs to be serviced. Specifically, it needs to be closed because it reached EOF. I explained this. Stop using select on a handle you're told is closed.

Re: Malfunctioning select() call on a FIFO
by zentara (Archbishop) on Jan 07, 2010 at 12:32 UTC

      Instead of :
      sysopen (FIFO, $icbmfifo, O_RDONLY|O_NONBLOCK);
      We use:

      sysopen (nullFile, $icbmfifo, O_RDONLY|O_NONBLOCK);<br/> sysopen (writeFile, $icbmfifo, O_WRONLY|O_NONBLOCK);<br/> sysopen (FIFO, $icbmfifo, O_RDONLY);<br/> close(nullFile);


      This makes the program wait for data on the FIFO.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://815950]
Approved by toolic
Front-paged by MidLifeXis
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2024-04-24 11:26 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found