olecs has asked for the wisdom of the Perl Monks concerning the following question:

Hello everyone, I am looking for a way to open an external program, feed a set of data to it and capturing the output in a variable, much like using @var = `extprog blah blaha` - extprog returns a set of values each time it gets input.

The problem is that I have a lot of data (several million sets of coordinates) and it takes a bit of time to open and initialize the external program each time, hence I would like to open the external program once and feed the data through a pipe into the program, I was imagining something like this: (note that the code might have errors, just showing for illustration)
open EXT,"|extprog";
while (<>){
  @data = conv_data($_);
  print EXT "$data[0] $data[1]\n";
}
close EXT;

[download]

Now, my external program will get this data and put it's data on stdout. The question is how do I get hold of/redirect the output, preferrably without resorting to temporary files.

rgds,
Ole C.

Update
I got this sorted now, thanks for the replies. I also got a couple of requests to share my final code and what I ended up doing in the end, so I will post it here.
The output from my external program was buffered, so I could not use open3 as the read/write deadlocked. Hence I ended up using brother merlyns suggestion which at first seemed like pure magic but which made more sense after reading up on -|
My final code for achieving what I wanted then looked like this:
$run =
"mapproject -F --D_FORMAT=\%lf -I --ELLIPSOID=HELMERT1906  -Jt31/30/1:
+1 -R31/40/30/39 --MAP_SCALE_FACTOR=1.0000 ";
open FH, "<depths.asc.to" or die "could not open asc file\n";
if ( $pid = open RESULTS, "-|" ) {
    # I am the original process, so read from <RESULTS> below
}
else {
    # I am the kid
    if ( $pid1 = open STDIN, "-|" ) {
        # I am still the kid, but my STDIN is now piped from the grand
+kid
        exec "$run";
    }
    else {    # I am the grandkid: be a data pump
    while (<FH>) {    
        print "$_\n";
      }
        exit 0;
    }
}
while (<RESULTS>) {
    print $_;
}

[download]

Replies are listed 'Best First'.
Re: output from an external program
by Zaxo (Archbishop) on Dec 01, 2005 at 13:48 UTC

    Take a look at IPC::Open3. It will do exactly what you want.

    After Compline,
    Zaxo

[reply]
      Thank you - that was exactly what I was looking for.
[reply]
        If you don't need interaction, consider a two-stage fork, which is safer than having one program both send and recieve with IPC::Open3, since you can never deadlock by being the wrong state. For example: 
        if (open RESULTS, "-|") {
  # I am the original process, so read from <RESULTS> below
} else {
   # I am the kid
  if (open STDIN, "|-") {
    # I am still the kid, but my STDIN is now piped from the grandkid
    exec "your", "big", "application";
    die "can't find big application: $!";
  } else { # I am the grandkid: be a data pump
    print "$_\n" for @input_data;
    exit 0;
  }
}
while (<RESULTS>) {
  ....
}

        [download]
        This double-fork data-pump is a nice pattern to keep in mind.

        -- Randal L. Schwartz, Perl hacker
        Be sure to read my standard disclaimer if this is a reply.

[reply]
[d/l]
Re: output from an external program
by davorg (Chancellor) on Dec 01, 2005 at 13:48 UTC

    Did you try reading the documentation for open?

    If the filename begins with '|' , the filename is interpreted as a command to which output is to be piped, and if the filename ends with a '|' , the filename is interpreted as a command which pipes output to us.

    Basically, your invented syntax is pretty much exactly what you'd need to do. You're just reading from the wrong filehandle in your while loop - you need to read from EXT.

    Oh, and you got the pipe symbol at the wrong end of the command.

    Update: Ignore me. I misread the question and didn't see the requirement to both read from and write to the filehandle. See IPC::Open3 (which is a standard module) as Zaxo suggests below.

    --
    <http://dave.org.uk>

    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg

[reply]
Re: output from an external program
by psychotic (Beadle) on Dec 01, 2005 at 16:17 UTC
    How about readpipe? 
    # Like this
my $multilineString = readpipe('extprog');
# or split on current $/
my @lines = readpipe('extprog')

    [download]
    Update: Fell in the same trap as davorg did. Ignore me likewise.
[reply]
[d/l]
[select]
Re: output from an external program
by neilwatson (Priest) on Dec 01, 2005 at 13:40 UTC
    What is the external program? There may be a Perl module osrprogram or even emulate it.

    Neil Watson
    watson-wilson.ca

[reply]
      The external program is a binary, written in C that does geographic projection shifts. There is no option but to use that particular program.
[reply]
Re: output from an external program
by tphyahoo (Vicar) on Dec 07, 2005 at 14:01 UTC
    Is there a reason you had 
    open EXT,"|extprog";
while (<>)

    [download]
    rather than 
    open EXT,"|extprog";
while (<EXT>)

    [download]
    When I experimented against open EXT,"|ls" the first version gave the directory listing, but then kept waiting for input from STDIN, until I gave ctrl-d, whereas the second printed the file listings and quit out, which is (naively?) the behavior I expect.

    Just curious!

    Also, thanks for updating with your final solution.

[reply]
[d/l]
[select]
      The reason was that I wanted to use data from STDIN and pipe it through extprog. When doing
      while (<>)

      [download]
      the loop will run as long as data kept coming from STDIN, and all data will be piped into extprog inside the loop.
      This as opposed to doing
      while (<EXT>)

      [download]
      which would require extproc to produce output for the loop to run. Since extproc requires input to produce output it would never run, atleast not the way my example was written.
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom