[user@host]> streaming_output_script.pl&
[download]

Well, I doubt you need to do that on unix. I certainly don't need to on Win32.

Given junk2.pl that looks like this (note this script won't terminate naturally for nearly 3 hours):

#! perl -slw
use strict;

$|++;

for( 1 .. 10000 ) {
    print scalar localtime;
    sleep 1;
    warn "$0 still running\n";
}
[download]

And calling that from this script using a piped open in the same manner as the OP uses:

#! perl -slw
use strict;

my $pid = open my $fh, 'perl.exe junk2.pl |' or die $!;
while( <$fh> ) {
    chomp;
    print localtime() . " Got: '$_'";
}
[download]

I get this output:

c:\test>junk3
Sat Aug  4 02:33:24 2007 Got: 'Sat Aug  4 02:33:24 2007'
junk2.pl still running
Sat Aug  4 02:33:25 2007 Got: 'Sat Aug  4 02:33:25 2007'
junk2.pl still running
Sat Aug  4 02:33:26 2007 Got: 'Sat Aug  4 02:33:26 2007'
junk2.pl still running
Sat Aug  4 02:33:27 2007 Got: 'Sat Aug  4 02:33:27 2007'
junk2.pl still running
Sat Aug  4 02:33:28 2007 Got: 'Sat Aug  4 02:33:28 2007'
Terminating on signal SIGINT(2)

c:\test>junk2.pl still running
junk2.pl still running
junk2.pl still running
junk2.pl still running
junk2.pl still running
junk2.pl still running
...
[download]

Which shows that the parent receives the output from the child as soon as it produces it. And I seriously doubt that the '&' is required under unix either.

Even if I leave stdout buffering enabled

#! perl -slw
use strict;

#$|++;

for( 1 .. 10000 ) {
    print scalar localtime;
    sleep 1;
    warn "$0 still running\n";
}
[download]

It takes a while longer (~3 minutes) before the child fills the output buffer and the parent starts to receive the output (in one huge flurry):

c:\test>junk3
junk2.pl still running
junk2.pl still running
...
165 identical lines elided
...
junk2.pl still running
junk2.pl still running
Sat Aug  4 02:38:30 2007 Got: 'Sat Aug  4 02:35:51 2007'
Sat Aug  4 02:38:30 2007 Got: 'Sat Aug  4 02:35:52 2007'
...
156 monotonically increasing lines elided
...
Sat Aug  4 02:38:31 2007 Got: 'Sat Aug  4 02:38:28 2007'
Sat Aug  4 02:38:31 2007 Got: 'Sat Aug  4 02:38:29 2007'
junk2.pl still running
junk2.pl still running
...
[download]

But it certainly doesn't have to wait 3 hours for the child to finish before it gets it.

So, unless java uses a huge output buffer, or the java program produces very little output, I doubt the problem is down to buffering either. And if it is, there is nothing the OP could do with buffering in the calling script to change it anyway.

So, it comes back to the question, what exactly did the OP mean when he said:

How can I get the STDOUT and STDERR from that pipe without waiting for it to complete...

And until he clarifies that, his SoPW is unanswerable.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

As I am new to the idea of "buffering" over a web interface, (most of these tools have been run from command line), I can explain this much for you - The java tests, if ran stand-alone, produces quite a bit of log-type output, and it prints all along the way while it is running.

Now when I run them using the piped open from the OP (original post? - still learning...) I get back to my original problem, the calling script will not print any of the pipe's output until the pipe command has completed.

I look at your first example though, do you think I am barking up the wrong tree to think the Perl caller is the problem? I have access to the java files, I will try tinkering with their output buffering and see if that yields any results.

Thanks!