Well, I doubt you need to do that on unix. I certainly don't need to on Win32.

Given junk2.pl that looks like this (note this script won't terminate naturally for nearly 3 hours):

#! perl -slw
use strict;

$|++;

for( 1 .. 10000 ) {
    print scalar localtime;
    sleep 1;
    warn "$0 still running\n";
}
[download]

And calling that from this script using a piped open in the same manner as the OP uses:

#! perl -slw
use strict;

my $pid = open my $fh, 'perl.exe junk2.pl |' or die $!;
while( <$fh> ) {
    chomp;
    print localtime() . " Got: '$_'";
}
[download]

I get this output:

c:\test>junk3
Sat Aug  4 02:33:24 2007 Got: 'Sat Aug  4 02:33:24 2007'
junk2.pl still running
Sat Aug  4 02:33:25 2007 Got: 'Sat Aug  4 02:33:25 2007'
junk2.pl still running
Sat Aug  4 02:33:26 2007 Got: 'Sat Aug  4 02:33:26 2007'
junk2.pl still running
Sat Aug  4 02:33:27 2007 Got: 'Sat Aug  4 02:33:27 2007'
junk2.pl still running
Sat Aug  4 02:33:28 2007 Got: 'Sat Aug  4 02:33:28 2007'
Terminating on signal SIGINT(2)

c:\test>junk2.pl still running
junk2.pl still running
junk2.pl still running
junk2.pl still running
junk2.pl still running
junk2.pl still running
...
[download]

Which shows that the parent receives the output from the child as soon as it produces it. And I seriously doubt that the '&' is required under unix either.

Even if I leave stdout buffering enabled

#! perl -slw
use strict;

#$|++;

for( 1 .. 10000 ) {
    print scalar localtime;
    sleep 1;
    warn "$0 still running\n";
}
[download]

It takes a while longer (~3 minutes) before the child fills the output buffer and the parent starts to receive the output (in one huge flurry):

c:\test>junk3
junk2.pl still running
junk2.pl still running
...
165 identical lines elided
...
junk2.pl still running
junk2.pl still running
Sat Aug  4 02:38:30 2007 Got: 'Sat Aug  4 02:35:51 2007'
Sat Aug  4 02:38:30 2007 Got: 'Sat Aug  4 02:35:52 2007'
...
156 monotonically increasing lines elided
...
Sat Aug  4 02:38:31 2007 Got: 'Sat Aug  4 02:38:28 2007'
Sat Aug  4 02:38:31 2007 Got: 'Sat Aug  4 02:38:29 2007'
junk2.pl still running
junk2.pl still running
...
[download]

But it certainly doesn't have to wait 3 hours for the child to finish before it gets it.

So, unless java uses a huge output buffer, or the java program produces very little output, I doubt the problem is down to buffering either. And if it is, there is nothing the OP could do with buffering in the calling script to change it anyway.

So, it comes back to the question, what exactly did the OP mean when he said:

How can I get the STDOUT and STDERR from that pipe without waiting for it to complete...

And until he clarifies that, his SoPW is unanswerable.

As I am new to the idea of "buffering" over a web interface, (most of these tools have been run from command line), I can explain this much for you - The java tests, if ran stand-alone, produces quite a bit of log-type output, and it prints all along the way while it is running.

Now when I run them using the piped open from the OP (original post? - still learning...) I get back to my original problem, the calling script will not print any of the pipe's output until the pipe command has completed.

I look at your first example though, do you think I am barking up the wrong tree to think the Perl caller is the problem? I have access to the java files, I will try tinkering with their output buffering and see if that yields any results.

Thanks!

I'm not a cgi person, but there are various things that could be causing the symptoms you are describing. You need to determine where the delays are occuring.

If the java program produces a steady stream of output when invoked via the command line, it probably isn't the source of your problem.

You could confirm that it isn't your perl cgi script, by invoking it directly from the command line and watching what happens. If it also produces a steady stream when invoked this way, then there are a couple of other places where the problem could lie. Eg.

• Depending upon the browser involved, and the html you are outputting, the browser could be refusing to format and display what it has received, until it gets the end tag for one or more elements.

  Some (maybe all?) browsers refuse to format tables until they know how many or how big the rows are.

  You could probably verify if this is the problem, by temporarily switching the content type header to text/plain. That should cause the browser to display the html and content as it arrives rather than waiting until it knows how to format it.

  If this is the problem, you may be able to avoid it by switching to using <p> or <br> formatting rather than (for example) a table.

• Depending upon the web server you are using, it may be buffering the output until it detects the end of the cgi task.

  A quick way of checking this would be to run the cgi script to completion, from the command line, having redirected stdout to a file. Then substitute a simple cgi script that slurps the file in one chunk, prints it to the browser in one chunk, and then sleeps for 5 minutes before exiting.

  If you see that output in the browser straight away, then the delay is in the browser. If you don't see the output for 5 minutes, the delay is in the server/cgi interface.

If you supply a better picture of what you are doing--what server, browser, html formatting etc--you are using, then one of the many experienced CGI people here will be probably be able to offer you better ways of determining the source of your problem, and probably a solution to it.

Ultimately, unless your web server is dedicated to running just this task, or just a few low-demand tasks, it's generally not a good idea to have a cgi script that takes hours to run. The usual solutions to this involve forking, but they generally don't work under Win32 unless you are also using cygwin. I do have an alternative to that, but you would still need to fix the above problem(s) first.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."