smackdab has asked for the wisdom of the Perl Monks concerning the following question:

I am looking for a way to process data using some small command-line utilities that each do 1 thing. (I run under windows).

I have a variable, @data, containing, well data ;)

I want to pass it to the first program, read the results. Then I want to pass the results to the second program and read the final answer. (it would be ideal to get to munge the @data between calling prog1 and prog2)

Kinda like: @ans = `getnames | sort`

But, I want to pass @data to getnames (which will do some work) and then have the results goto sort (which will do more work), then back in my Perl program.

One option might be to use 1 or more temporary files:
write @data to temp1.dat
execute `type temp1.dat | getnames > temp2.dat`
(would be ideal to muncge @data here...) execute `type temp2.dat | sort > temp3.dat`
@ans = `type temp3.dat`

I had originally thought stdin and maybe Open2 would work, but, 'kwoff' pointed out that Open2, will block for programs like "sort".

I do have access to the "c" source of the utilites. So I could go in and make sure that I read all stdin and close it before I try to write to stdout. (I think it does this already...from what I can tell ;-)

One cool thing about Unix is all of those small utilities that you can combine into powerfull functions...I hope to emulate this, instead of using temp files that need to be cleaned up...

This might be hard or easy, but please don't assume, I know what I am doing ;-) and thanks for any ideas

Replies are listed 'Best First'.
Re: calling mult files
by jryan (Vicar) on Dec 03, 2001 at 08:21 UTC

    Well, since you are Windows, IPC::Shareable and Mmap are out of the question. While the temp file solution will work, it will be hideously slow when dealing with large amounts of data.

    One solution you can try is to use Storable to serialize data and send it through sockets where your programs can pick it up. However, that might be tricky. However, as your luck may have it, I am finishing up on a module (that I have been working on for quite some time) that does exactly that :) If you don't mind waiting a few days, you might want to take a look at it when it is done to see if it does what you need.

Re: calling mult files
by faerloche (Sexton) on Dec 03, 2001 at 18:05 UTC
    Something like this might do what you want:
    my $data_str = join('', @data); my $results1 =`command1 $data_str`; my $results2 =`command2 $results1`; print $results2;
    Using backticks will execute the command (command1/command2) and return the output. It'll also interpolate the variables so you can pass in the results of the previous command :-)
Re: calling mult files
by smackdab (Pilgrim) on Dec 04, 2001 at 09:41 UTC
    Well...since you have access to the source code, why don't you just combine the 2 'c' utilities?

    People did program in other languages before Perl ;-)