There is pipe. Pipe is lower level than shell pipes so is more flexible. It is possible to set up arbitrary networks of processes with pipe, fork, and select. The complexity of such things prevents their use, but that is not a limit of Perl.

It is pretty easy to set up a coprocess which you can print stuff to, and read back the result. Skeletal version:

my ($pin,$pout,$cin,$cout,%kid);
{
    pipe $pin, $cout;
    pipe $cin, $pout;
    my $cpid = fork;
    die $! unless defined $cpid;
    $kid{$cpid} = 1, last if $cpid; # parent
    close $pin or die $!;   # in child to end of block
    close $pout or die $!;
    while (<$cin>) {
        # do filter-like stuff to $_
        print $cout $_;
    }
    exit 0;
}
# parent
close $cin or die $!;
close $cout or die $!;
# print to $pout and read from $pin, maybe in a select loop
# depends on expected behavior of the child.
delete $kid{wait()} while %kid;
[download]

I'm quite familiar with streams, and with HOP's treatment of them. I think it would be exceedingly awkward to implement the example in my OP using streams. (For one thing, since I am sorting in the last step, I have to fully "actualize" each stream, which means that they become garden-variety (finite) linked lists, each one of them resident in memory.) What I'm talking about has much more to do with iterators, which Dominus also covers extensively in HOP.

the lowliest monk

Well, you can't sort infinite streams. I haven't read HOP, but usually streams and iterators are the same thing. Lazy lists, on the other hand, are nothing but memoized iterators. You can sort iterators without storing the entire thing in memory, but it requires multiple passes through contents of the iterator (N passes for the bubble sort, although you can trade this off for increased memory usage. That is you can do it in N/2 passes, if your bubble sort floats two items to the top at a time).

You do know about open("-|") and open("|-"), don't you?

However, I don't like to pipe data through real pipes between processes. I rather like real coroutines that pass values to each other, not just text through a pipe. Perl is not really good in this, because it doesn't have continuations and coroutines, and it's very difficult to add them to the perl core now. If you want that, you can use one of

• python which has coroutines and transparent lazy lists based on them,
• perl6 which will have them too but it's not ready yet,
• scheme has continuations built in, and some scheme implementations and libraries have even more support
• ruby, which has a callback-based iterator modell in its core classes; but also continuations and some libraries based on it
• lua which has coroutines

  # chain_lines takes a list of coderefs
  # The first sub in the chain is called with no input; subsequent one
+s are called
  # with $_ set to the line from the previous sub; the iterator return
+s one line of
  # output from the last sub in the chain. If a sub yields undef, proc
+essing
  # restarts at the first sub. When the first sub yields undef, the it
+erator is done.
  my $i = chain_lines sub { <$in> },
                      sub { /^42/ ? $_ : undef },    # grep
                      sub { join "\t", (split /\t/)[1,3,8] };
  # all_lines expands the iterator into a list of lines
  sort all_lines($i);
[download]

grep_lines {/^42/}
[download]

If you're going to write a grep iterator, have the return value be an empty list on element failure. This allows failures to be removed from the list instead of just being substituted with undefined values. In fact, grep is just a special case of map.

grep /^42/
map { /^42/ ? $_ : () }
sub { /^42/ ? $_ : () }
[download]

undef is a special value in the scheme I've proposed. All results are taken in scalar context (assigned to $_), since this is a line-piping scheme. If the grep step returns undef, the iterator goes back to the first sub to try a new line. undefs do not show up in the output stream.

You could use an empty list as your no-line-returned indicator, but it would require you evaluating the subs in list context and then turning the returned value into a scalar afterward. That would be problematic for some common pipe actions, like reading one line at a time from a file (in list context, the whole file would be read).

---

Caution: Contents may have been coded under pressure.

mdb-export -d , file.mdb tablename | gzip -9 -c > file.csv.gz
[download]

use IPC::Open2;
use strict;

opendir DIR, ".";
my @bancos = grep { /.mdb$/ } readdir DIR;
closedir DIR;

foreach my $db (@bancos) {
        chomp $db;
        print "$db\n";
        open TABLES, "mdb-tables -1 $db |" || die $!;
        my @tables = <TABLES>;
        close TABLES;
        my $dbdir = $db;
        $dbdir =~ s/\.mdb$//;
        mkdir $dbdir || die $!;
        foreach my $table (@tables) {
                chomp $table;
                open('MDBEXPOUT', "mdb-export -d , '$db' '$table' |") 
+|| die $!;
                $table =~ s/\W/_/g;
                open('OUTFILE', ">$dbdir/$table.csv.gz") || die $!;
                open2('>&OUTFILE', '<&MDBEXPOUT', "gzip", "-9", "-c") 
+|| die $!;
                wait;
                close MDBEXPOUT;
                close OUTFILE;
                print "$dbdir/$table\n";
        }
}
[download]