Reading partial lines/strings from a pipe

slightly72 has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

I am trying to write some Perl code to communicate with a server. The issue here is that the server replies with strings (i.e. not full lines) -- for example, a query for some port number is answered with "XXXX" and not "XXXX\n" (or any other line/string terminator). I am using open2 to launch the server:

use IO::Handle;<br>
my ($wrt, $rdr) = (IO::Handle->new(), IO::Handle->new());
my $procid = open2($wrt, $rdr, $server_command_line);
[download]

Then at some point I write the command to the stdin of the server (now piped through $wrt). The problem is with reading from the stdout of the server, piped through $rdr.
If I try:
my $reply = <$rdr>;
the program deadlocks since a full line is expected but never arrives.

I've tried to read character by character from the file (it's not a large performance penalty since the replies are quite short, at most 8-10 chars long) -- however, I don't know when to stop reading these characters, if I'm trying to read over the length of the reply, I get deadlocked again.

The replies are in no particular format or fixed length.

Ideally, I would like to handle the reply with something like:

while (1) {
  if (there_is_something_to_read($rdr)) {
    $reply = get_everything_from_the_pipe($rdr);
    last;
  }
}
[download]

I have looked into File::Tail, but that seems to work only for regular files and not pipes. I've looked also into using threads, as per a reply to a similar problem on perlmonks, but again the issue is that that solution worked with 'records' (i.e. usually lines).
The server is proprietary software, so I can't get into the sources to modify the replies.

Right now, I'm at a loss for a solution, so any help is really appreciated. Oh, and if the solution is not too OS dependent (my script should work on both Linux and Windows) it would be terrific.

Thank you in advance,
Tibi

Comment on Reading partial lines/strings from a pipe Select or Download Code

Replies are listed 'Best First'.
Re: Reading partial lines/strings from a pipe by philcrow (Priest) on Oct 10, 2007 at 18:06 UTC
When working with raw bytes, it pays to use the read method. It will block until something is available, then read up to a specified number of bytes: `# from Programming Perl my $buffer; while ( read $your_handle, $buffer, 1024 ) { # process buffer here }` [download] That will read up to 1K from the handle, blocking if nothing is there, but returning whatever is there (the first 1K of it anyway). The Perl Cookbook is an excellent reference for doing this sort of work. Phil The Gantry Web Framework Book is now available.	[reply] [d/l]
Re^2: Reading partial lines/strings from a pipe by slightly72 (Initiate) on Oct 10, 2007 at 18:24 UTC
Phil, thank you very much for the pointer and the book reference, it got me in the right direction. The final solution is almost like the one you suggested: `my $buffer = ''; $rdr->blocking(0); while ($buffer eq '') { $rdr->read($buffer,1024); }` [download] $rdr->read is just an OO wrapper for read. The key to making this work was to use $rdr->blocking(0), which makes the pipe reading non-blocking. Otherwise a deadlock occurs -- maybe because the read command blocks while waiting for 1k to be filled, but not sure. Thank you, Tibi	[reply] [d/l]
Re: Reading partial lines/strings from a pipe by kyle (Abbot) on Oct 10, 2007 at 18:15 UTC
for the `there_is_something_to_read` functionality you're looking for, have a look at select (second form) and IO::Select. Note that when you're using these, you need to use sysread to get your data.	[reply] [d/l]