how to increase input buffer size in perl?

shooter714 has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,
So I have a two scripts that does a lot of computation for separate things: one in perl and one in IDL. Bear with me if you all don't IDL; I think it's a perl issue. Well, I need them to interface with each other so that I can pass two very large perl arrays into IDL which can process them and spit out an output back to me in perl. I do this using IPC:Open2 and the excerpt of my code is below:

use IPC::Open2
$progName = "idl";
open(READIDL, WRITEIDL, $progName)

print WRITEIDL <<"ENDIDL"
arr1 ='@array1'
arr2 ='@arrat2'
program, arr1, arr2, data
print, data
ENDIDL
$data = <READIDL>
chomp($data)
[download]

Now, program is not an IDL batch file. It is a procedure. So I must pass these arrays into it and data is returned as long as i do a print, data command at the IDL prompt before ENDIDL, and then I use the READIDL filehandle to retrieve it.

The problem once I compile and run my perl script is:
% Input line is too long for input buffer of 32766 characters. Broken pipe
array1 and array2 are large string arrays and take up a lot of memory. I was wondering if there is some way I can increase the buffer size so that perl can pass them into IDL without breaking the pipe. I'm using perl version 5.8.8. Thanks for the help.

Comment on how to increase input buffer size in perl? Download Code

Replies are listed 'Best First'.
Re: how to increase input buffer size in perl? by BrowserUk (Patriarch) on Mar 31, 2010 at 19:04 UTC
Input line is too long for input buffer of 32766 characters. That comes from IDL itself. It is probably caused by: `arr1 ='@array1'` [download] Which if @array1 contains a large number of values, will come out as one very long line. I don't know IDL syntax, but you might work around it by breaking the array assignment across multiple lines something like: `my $array1 = ''; $array1 .= @array1[ $_ .. $_ + 10 ] . "\n" for map $_* 10, 0 .. $#arra +y1 / 10; ... arr1 = '$array1'` [download] You also might look at PDL. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "I'd rather go naked than blow up my ass"	[reply] [d/l] [select]
Re^2: how to increase input buffer size in perl? by Anonymous Monk on Mar 31, 2010 at 21:10 UTC
Thanks BrowserUk. That must be it. But I don't get how that line of code you gave works? Could you explain?	[reply]
Re^3: how to increase input buffer size in perl? by BrowserUk (Patriarch) on Mar 31, 2010 at 21:47 UTC
The notion was to break up the long line by inserting newlines every 10 values. But, if IDL accepts that, then it would be far easier to just put one value per line: `my $array1 = join "\n", @array1; ... arr1 = '$array1' ...` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "I'd rather go naked than blow up my ass"	[reply] [d/l]
Re: how to increase input buffer size in perl? by Anonymous Monk on Mar 31, 2010 at 17:54 UTC
Use strict warnings and diagnostics or die you're using open when you should be using open2 "input line is too long" is a shell error (cmd/sh/bash...) not perl (probably because youre not using open2)	[reply]
Re^2: how to increase input buffer size in perl? by shooter714 (Initiate) on Mar 31, 2010 at 18:17 UTC
Oh, that is a typo. I didn't copy and paste from my script but I AM using open2. Still have any ideas anyone?	[reply]
Re^3: how to increase input buffer size in perl? by ikegami (Patriarch) on Mar 31, 2010 at 18:28 UTC
Could you turn that into runnable code, please? It's not clear where the message comes from. I doubt it's from Perl. Finding the cause (Perl or otherwise) is the first step in finding a solution.	[reply]
Re^4: how to increase input buffer size in perl? by Anonymous Monk on Mar 31, 2010 at 18:35 UTC
Re^5: how to increase input buffer size in perl? by ikegami (Patriarch) on Mar 31, 2010 at 18:49 UTC
Re^3: how to increase input buffer size in perl? by shooter714 (Initiate) on Apr 07, 2010 at 03:46 UTC
Yeah, the error has to do with how many arguments idl can accept. It caps out at 32766 character length. Instead I had to just write the array to a file and read that file inside the IDL script. Apparently you can't break up the array into separate lines either. IDL will throw an error.	[reply]