The problem is introduced by the OS handling of pipes.

I've seen this said here before--and I've even believed it--but it's not the case. Take the following example (wrapped for posting):

perl -e"$Win32::Sleep(100) and print qq[$_ ...] for 1 .. 100"
 | perl -ne"print qq[\r'$_']"
[download]

One perl process piped to another by the OS. The attempt is to reflect the progress indicator produced by the first process using the second. Needless to say, it doesn't work and there is nothing for 10 seconds, then like London busses, all the output comes along at once.

But, where are the delays/buffering?

If you try the first process on it's own

perl -e"Win32::Sleep(100) and print qq[$_ ...] for 1 .. 100"
[download]

You'll again wait 10 seconds before seeing anything, and then get it all at once. So one level of buffering is in the originating process. Easily fixed using $|++.

perl -e"$|++; Win32::Sleep(100) and print qq[$_ ...] for 1 .. 100"
[download]

And sure enough, we see new output appear every 1/10th of a second for 10 seconds.

So, now try to read and reflect that via a pipe:

perl -e"$|++; Win32::Sleep(100) and print qq[$_ ...] for 1 .. 100" 
 | perl -ne"print qq[\r'$_']"
[download]

And once again, nothing, nothing, nothing, Bang! Must be the pipe that's the problem right?

Well no. The second perl process is using the default $INPUT_LINE_SEPARATOR, so it doesn't give the perl program any input until it sees a newline. But the first process doesn't produce any newlines. Ever. So the second doesn't give the program any input until it see EOF.

So how about we set the $IRS for the second process:

perl -e"$|++; Win32::Sleep(100) and print qq[$_ ...] for 1 .. 100" 
| perl -e"BEGIN{ $/='...'}" -ne"print qq[\r'$_']"
[download]

Same deal. All the output comes in one lump after 10 seconds. It has to be an OS problem right!

But still no. The second perl process is still buffering its output. We are still not adding any newlines anywhere, so it again waits for us to tell it when to flush the output.

So, disable stdout buffering on the second process also:

perl -e"$|++; Win32::Sleep(100) and print qq[$_ ...] for 1 .. 100" 
 | perl -e"BEGIN{ $|++; $/='...'}" -ne"print qq[\r'$_']"
'100 ...'
[download]

And sure enough, this time we get the effect we are after. The progress indicator rolls over once every 1/10th of a second, overlaid in-place because the second Perl process added \rs.

Note. The piping, as performed by the OS didn't change. All the changes are with the originating and receiving processes. In this case Perl. If they are are set up to communicate directly, without buffering, then they can.

What this shows is that any output from the first process is immediately available to the second. All the buffering is occuring with the processes themselves, not within the OS/pipe. The problem is not in the OS piping, but within the defaults set by the Perl processes.

It may well be that under other OSs, the OS disables buffering on stdin/stdout within the processes it starts, when it connects them via pipes. If PerlIO chose to emulate that, when it is printing output to, or receiving input from a pipe, it could.