I think you've misunderstood what I'm after.

I want to execute multiple commands, piped together, but with error checking, STDERR capture, and the like, from within my scripts, hence the comments about the IPC:: modules and piped open()s, not make single scripts behave nicely as part of a pipeline.

Cheers.

BazB.

Update to pg's update: the suggestion to read the output from the first command into an array will not work.
That data could be upto ~30 million lines or ~50Gb.
That's the whole point of pipes - you don't have to be able to store the whole dataset in memory, nor waste time writing intermediate stages to disk.

The potential size of the input is also why I use while loops in my current code, although read() would probably be more efficient, since the data consists of fixed-length records.

Doing this in the shell directly might be easier, but the benefits of using Perl for building the application are more of an issue.

Update 2: Ah ha! Me thinks pg(++!) has cracked it.
pipe() seems to be the way to go. I'm rather surprised that I'd not come across it before.

BazB and I had some very interesting discussions on and off, in the chat room, and through MessageBox, now we come to an agreement that, the pipe() function introduced in perlipc would be a good solution to connect IO handlers between processes.

He suggested me to post a reply to complete this thread, and I am doing so now. A piece of sample attached:

use IO::Handle;

use strict;

$| ++;
pipe(PARENT_READER, CHILD_WRITER);
pipe(CHILD_READER, PARENT_WRITER);
PARENT_WRITER->autoflush(1);
CHILD_WRITER->autoflush(1);

my $pid;
if ($pid = fork()) {
    close(CHILD_READER);
    close(CHILD_WRITER);
    my $buffer;
    print PARENT_WRITER 1;
    while (1) {
        sysread(PARENT_READER, $buffer, 100);
        print "parent revd: $buffer, and reply with ", $buffer + 1, "\
+n";
        sleep(1);
        print PARENT_WRITER $buffer + 1;
    }
} else {
    close(PARENT_READER);
    close(PARENT_WRITER);
    my $buffer;
    while (1) {
        sysread(CHILD_READER, $buffer, 1000);
        print "child revd: $buffer, and reply with ", $buffer + 1, "\n
+";
        sleep(1);
        print CHILD_WRITER $buffer + 1;
    }
}
[download]

But if you want your script to look through the output while it's between the two processes to check for errors and such, you do need the loop with the read/write. If point each process at one end of a pipe you created, it will go on without your involvement. Maybe you can use the efficient pipe for stdout/stdin but still monitor stderr within the script.

I wrote a filter to act as a sniffer for a specific network protocol. I wrote code in Win32 to use IO completion ports to efficiently move the buffer from one to the other without copying it, and also passed it to an embedded Perl interpreter as a "tee" in the middle. The Perl could do its fine job of parsing the stuff and presenting it to me, but the processes piped efficiently as long as I didn't select the option for modification of the data stream by the Perl code (that is, report only). I used the modification ability to introduce errors or otherwise test things that my well-behaved program never did.

—John