IPC3 buffer limit problem

I was toying with using IPC3 to run bc as an educational example. See this nodeIPC3 buffer problemAnyways, I came up with some code to beat the 4k buffer limit, to give unlimited bc output. It's a hard to solve problem, do a groups.google search for "IPC perl buffer size limit". Anyways, here is what I came up with. All comments and criticisms welcome. It's how I learn :-) The basic trick I use is to use a do loop to keep reading the buffer until the buffer is less than full. Then I display the concantenated output string. The 4k buffer is 4060 bytes on my linux machine.

#!/usr/bin/perl
use warnings;
use strict;
use IPC::Open3;
require 'sys/ioctl.ph';

# I tested this with  
#  "q @ 2" to generate errors 
# "123^12345" to generate big output 
# "123^23456" to generate huge output 

#interface to "bc" calculator 
my $pid = open3(\*WRITE, \*READ,\*ERROR,"bc");
            #if \*ERROR is false, STDERR is sent to STDOUT  

while(1){

my($error,$answer,$rsize,$esize,$errortot,$answertot)=('','',0,0,'',''
+);

print "Enter expression for bc, i.e. 2 + 2\n";
chomp(my $query = <STDIN>);
if(length($query) == 0){next} #removes empty input 

#send query to bc 
print WRITE "$query\n";

# this do loop waits and eliminates need for a delay 
# to wait for bc to  output 
# It is NOT limited to 4060 bytes from bc, it keeps reading 
# until a semi-full buffer is sent, which signals end of output 

  do {
      #see which filehandles have output from perldoc -q filehandle 
      $esize = pack("L", 0);
      ioctl(\*ERROR, FIONREAD(), $esize) or die "Couldn't call ioctl: 
+$!\n";
      $esize = unpack("L", $esize);
      print "esize-> $esize\n" unless ($esize < 1);

      $rsize = pack("L", 0);
      ioctl(\*READ, FIONREAD(), $rsize) or die "Couldn't call ioctl: $
+!\n";
      $rsize = unpack("L", $rsize);
      print "rsize-> $rsize\n" unless ($rsize <1);

     #get the output from bc 
     if($esize > 0){sysread(ERROR,$error,$esize); $errortot = $errorto
+t.$error}
     if($rsize > 0){sysread(READ,$answer,$rsize); $answertot = $answer
+tot.$answer}
    } until(($esize > 0)or(($rsize > 0)and($rsize < 4060)));


if(length($errortot) > 0){ print "\e[1;31m ERROR-> $errortot \e[0m \n"
+}
if(length($answertot) > 0){print "Output->$query = $answertot\n"}

}
[download]

Comment on IPC3 buffer limit problem Download Code

Replies are listed 'Best First'.
•Re: IPC3 buffer limit problem by merlyn (Sage) on Oct 19, 2003 at 23:05 UTC
By providing tiny inputs, you're not hitting the real problem. If you need to pump more than 4K into the input buffer but can't start reading the output until after that, and the output hits 4K, you're toast. Certainly, if you could use async or pseudo-async handlers, you've got it made. See the way that POE deals with child running processes, especially POE::Wheel::Run. -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply]
Re: IPC3 buffer limit problem by zentara (Cardinal) on Oct 20, 2003 at 13:58 UTC
Merlyn said: If you need to pump more than 4K into the input buffer but can't start reading the output until after that, and the output hits 4K, you're toast. Yeah, I realized that after I posted it. The "edge case" of where the last buffer filled is 0 or 4060 is a problem. More thinking is needed.	[reply]
Re: IPC3 buffer limit problem by zentara (Cardinal) on Jan 03, 2006 at 15:12 UTC
Update: Jan-03-2006 just tried to run this script again on my newer linux system with a 2.6.x kernel and Perl 5.8.7. Anyways, there seems to be a difference in the way pipes work, and the above script no longer works. Below is another version that does work. It includes an output-to-file, so you can compare the output to bc running in a terminal. #!/usr/bin/perl use warnings; use strict; use IPC::Open3; require 'sys/ioctl.ph'; use FileHandle; # for testing output to compare to # bc's output, needed for autoflush $\| = 1; #to write to a file to compare to bc's output open (OUT, "> $0.out") or die "$!\n"; OUT->autoflush(1); # I tested this with # "q @ 2" to generate errors # "123^12345" to generate big output # "123^23456" to generate huge output #interface to "bc" calculator my $pid = open3(\WRITE, \READ,\ERROR,"bc"); #if \ERROR is false, STDERR is sent to STDOUT while(1){ my($error,$answer,$rsize,$esize,$errortot,$answertot)=('','',0,0,'','' +); print "Enter expression for bc, i.e. 2 + 2\n"; chomp(my $query = <STDIN>); if(length($query) == 0){next} #removes empty input #send query to bc print WRITE "$query\n"; #this do loop waits and eliminates need for a delay to let bc output #It is NOT limited to 4060 bytes from bc, it keeps reading #until a semi-full buffer is sent, which signals end of output do { #see which filehandles have output from perldoc -q filehandle $esize = pack("L", 0); ioctl(\ERROR, FIONREAD(), $esize) or die "Couldn't call ioctl: +$!\n"; $esize = unpack("L", $esize); print "esize-> $esize\n" unless ($esize < 1); $rsize = pack("L", 0); ioctl(\READ, FIONREAD(), $rsize) or die "Couldn't call ioctl: $ +!\n"; $rsize = unpack("L", $rsize); print "rsize-> $rsize\n" unless ($rsize <1); #get the output from bc if($esize > 0){sysread(ERROR,$error,$esize); $errortot = $errorto +t.$error} # needed this line on older linuxes and Perl, the pipes worked diff +erent # if($rsize > 0){sysread(READ,$answer,$rsize); $answertot = $answe +rtot.$answer} # on newer kernels 2.6.x and Perl 5.8.7, this works if($rsize > 0){sysread(READ,$answer,4096); $answertot = $answerto +t.$answer} } until( ($esize > 0) or ( ($rsize > 0) and ($rsize < 4096) ) ); if(length($errortot) > 0){ print "\e[1;31m ERROR-> $errortot \e[0m \n" +} if(length($answertot) > 0){print "Output->$query = $answertot\n" } print OUT $answertot; } close OUT; waitpid($pid, 1); # It is important to waitpid on your child process, # otherwise zombies could be created. [download] I'm not really a human, but I play one on earth. flash japh	[reply] [d/l]