output from an external program

olecs has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone, I am looking for a way to open an external program, feed a set of data to it and capturing the output in a variable, much like using @var = `extprog blah blaha` - extprog returns a set of values each time it gets input.

The problem is that I have a lot of data (several million sets of coordinates) and it takes a bit of time to open and initialize the external program each time, hence I would like to open the external program once and feed the data through a pipe into the program, I was imagining something like this: (note that the code might have errors, just showing for illustration)

open EXT,"|extprog";
while (<>){
  @data = conv_data($_);
  print EXT "$data[0] $data[1]\n";
}
close EXT;
[download]

Now, my external program will get this data and put it's data on stdout. The question is how do I get hold of/redirect the output, preferrably without resorting to temporary files.

rgds,
Ole C.

Update
I got this sorted now, thanks for the replies. I also got a couple of requests to share my final code and what I ended up doing in the end, so I will post it here.
The output from my external program was buffered, so I could not use open3 as the read/write deadlocked. Hence I ended up using brother merlyns suggestion which at first seemed like pure magic but which made more sense after reading up on -|
My final code for achieving what I wanted then looked like this:

$run =
"mapproject -F --D_FORMAT=\%lf -I --ELLIPSOID=HELMERT1906  -Jt31/30/1:
+1 -R31/40/30/39 --MAP_SCALE_FACTOR=1.0000 ";
open FH, "<depths.asc.to" or die "could not open asc file\n";
if ( $pid = open RESULTS, "-|" ) {
    # I am the original process, so read from <RESULTS> below
}
else {
    # I am the kid
    if ( $pid1 = open STDIN, "-|" ) {
        # I am still the kid, but my STDIN is now piped from the grand
+kid
        exec "$run";
    }
    else {    # I am the grandkid: be a data pump
    while (<FH>) {    
        print "$_\n";
      }
        exit 0;
    }
}
while (<RESULTS>) {
    print $_;
}
[download]

Comment on output from an external program Select or Download Code

Replies are listed 'Best First'.
Re: output from an external program by Zaxo (Archbishop) on Dec 01, 2005 at 13:48 UTC
Take a look at IPC::Open3. It will do exactly what you want. After Compline, Zaxo	[reply]
Re^2: output from an external program by olecs (Scribe) on Dec 01, 2005 at 14:00 UTC
Thank you - that was exactly what I was looking for.	[reply]
Re^3: output from an external program by merlyn (Sage) on Dec 01, 2005 at 14:44 UTC
If you don't need interaction, consider a two-stage fork, which is safer than having one program both send and recieve with IPC::Open3, since you can never deadlock by being the wrong state. For example: `if (open RESULTS, "-\|") { # I am the original process, so read from <RESULTS> below } else { # I am the kid if (open STDIN, "\|-") { # I am still the kid, but my STDIN is now piped from the grandkid exec "your", "big", "application"; die "can't find big application: $!"; } else { # I am the grandkid: be a data pump print "$_\n" for @input_data; exit 0; } } while (<RESULTS>) { .... }` [download] This double-fork data-pump is a nice pattern to keep in mind. -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply] [d/l]
Re: output from an external program by davorg (Chancellor) on Dec 01, 2005 at 13:48 UTC
Did you try reading the documentation for open? If the filename begins with '\|' , the filename is interpreted as a command to which output is to be piped, and if the filename ends with a '\|' , the filename is interpreted as a command which pipes output to us. Basically, your invented syntax is pretty much exactly what you'd need to do. You're just reading from the wrong filehandle in your while loop - you need to read from EXT. Oh, and you got the pipe symbol at the wrong end of the command. Update: Ignore me. I misread the question and didn't see the requirement to both read from and write to the filehandle. See IPC::Open3 (which is a standard module) as Zaxo suggests below. -- <http://dave.org.uk> "The first rule of Perl club is you do not talk about Perl club." -- Chip Salzenberg	[reply]
Re: output from an external program by psychotic (Beadle) on Dec 01, 2005 at 16:17 UTC
How about `readpipe`? `# Like this my $multilineString = readpipe('extprog'); # or split on current $/ my @lines = readpipe('extprog')` [download] Update: Fell in the same trap as davorg did. Ignore me likewise.	[reply] [d/l] [select]
Re: output from an external program by neilwatson (Priest) on Dec 01, 2005 at 13:40 UTC
What is the external program? There may be a Perl module osrprogram or even emulate it. Neil Watson watson-wilson.ca	[reply]
Re^2: output from an external program by olecs (Scribe) on Dec 01, 2005 at 13:47 UTC
The external program is a binary, written in C that does geographic projection shifts. There is no option but to use that particular program.	[reply]
Re: output from an external program by tphyahoo (Vicar) on Dec 07, 2005 at 14:01 UTC
Is there a reason you had `open EXT,"\|extprog"; while (<>)` [download] rather than `open EXT,"\|extprog"; while (<EXT>)` [download] When I experimented against `open EXT,"\|ls"` the first version gave the directory listing, but then kept waiting for input from STDIN, until I gave ctrl-d, whereas the second printed the file listings and quit out, which is (naively?) the behavior I expect. Just curious! Also, thanks for updating with your final solution.	[reply] [d/l] [select]
Re^2: output from an external program by olecs (Scribe) on Dec 08, 2005 at 09:43 UTC
The reason was that I wanted to use data from STDIN and pipe it through extprog. When doing `while (<>)` [download] the loop will run as long as data kept coming from STDIN, and all data will be piped into extprog inside the loop. This as opposed to doing `while (<EXT>)` [download] which would require extproc to produce output for the loop to run. Since extproc requires input to produce output it would never run, atleast not the way my example was written.	[reply] [d/l] [select]


Perl: the Markov chain saw
	PerlMonks