Autoflush and web browsers (was: Clever autoflush detail)

talexb has asked for the wisdom of the Perl Monks concerning the following question:

I received a query from a client about how to get autoflush working, so cobbled together a simple example. First, I did it in shell,

#!/bin/sh

echo "Content-type: text/plain";
echo

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do

  echo "Value is $i";
  sleep 1;
done
[download]

This worked fine (it output a line every second), and I translated it into Perl.

#!/usr/bin/perl -w

$|=1;

use CGI qw/:standard/;

{
    my $cgi = CGI->new;
    print $cgi->header('text/plain');

    print "Autoflush is currently ".($|?"on":"off").".\n";
    for ( 1..5 ) {

      print "Value is $_.\n";

      sleep 1;
    }
    print "Autoflush is currently ".($|?"on":"off").".\n";
}
[download]

This didn't work. Nothing was output until the script finished. I was stumped. I search Perlmonks and found lots of information on autoflush questions, nodes Autoflush $|?, Autoflushing Laser printer, How to get explicity the autoflush var ($|) of a TIED HANDLER?, Autoflushing revisited., Simple autoflush module, OUTPUT_AUTOFLUSH, autoflush and 'DESTROY', autoflushing.. and When to use Autoflush. Nothing solved my problem.

Finally, over on IRC, I put my question and got a response from Caelum -- use PRE tags.

#!/usr/bin/perl -w

$|=1;

use CGI qw/:standard/;

{
    my $cgi = CGI->new;
    print $cgi->header('text/html');

    print pre("Autoflush is currently " . ($|?"on":"off") );
    for ( 1..5 ) {

      print pre("Value is $_.</pre>");

      sleep 1;
    }
    print pre("Autoflush is currently " . ($|?"on":"off") );
}
[download]

Well, to cut a long story short, it works. But why?

Shouldn't the autoflushing (happening down at layer 2 or 3) be immune to the presentation (up at layer 7)? Is this a Perl oddity?

What is going on?

Update: As fenLisesi points out, the closing pre tag in the 'Value is' print statement shouldn't be there, but I won't edit the original node to reflect that..

Update 2: I don't mind changing the title of this node to Autoflush and web browsers, but that's naming the thread after the fact -- after we know (or suspect) that it's the interaction with the browser that's causing the odd behaviour. In addition, it looks like most of the discussion on this thread is over (at the time of writing, the most recent reply to the root was shmem's dated 18 hours ago), so this consideration may not be necessary.

Update 3: Brother benizi makes a very good point -- retitling the node would be an aid to future searches, therefore it's a Good Thing. I've held off voting on the consideration or making a title change, but now I'm convinced.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Comment on Autoflush and web browsers (was: Clever autoflush detail) Select or Download Code

Replies are listed 'Best First'.

Re: Clever autoflush detail
by shmem (Chancellor) on Jun 20, 2007 at 21:12 UTC

    print "Content-type: text/plain;\r\n\r\n";
[download]

and

    print "Content-type: text/plain; charset=ISO-8859-1\r\n\r\n";
[download]

give different results - no matter which of shell, perl, with or without CGI is involved. For the latter, the browser seems to ~~gather all its input first~~ read in blocks, presumably to apply conversions ~~on the complete text~~ to each. (tested with Firefox only).

Try appending "\0" x 1024 to each line using CGI with print $cgi->header("text/plain").

Another way to confirm that unbuffered IO is turned on with CGI is snarfing the packets with eg wireshark.

...another update: even the trailing ; after text/plain makes a difference. If you are not outputting that, the browser buffers.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

[reply]
[d/l]
[select]

Re: Clever autoflush detail
by BrowserUk (Patriarch) on Jun 20, 2007 at 18:46 UTC

I don't do cgi, but surely the problem (being fixed by using PRE tags), is not at the output end, but at the browser end?

Normally, html ignores CR/LF. So, the browser cannot display the line until it gets something (like: BR or P) that tells it that this 'line' is complete and can be formatted. Using the PRE tags means that CR/LF is respected, so the browser can display the line as soon as it sees those characters.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

[reply]

Re^2: Clever autoflush detail

by talexb (Chancellor) on Jun 20, 2007 at 18:51 UTC

Normally, html ignores CR/LF. So, the browser cannot display the line until it gets something (like: BR or P) that tells it that this 'line' is complete and can be formatted.

Right, but I made a point of setting the content type to be text/plain (that is, specifically not to be HTML), and that worked correctly using the shell script, but not using the Perl script. That's why I'm confused.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

[reply]

Re^3: Clever autoflush detail

by Joost (Canon) on Jun 20, 2007 at 19:28 UTC

from your code: print $cgi->header('text/html');

"What should it profit a man, if he should win a flame war, yet lose his cool?"

[reply]
[d/l]

Re^4: Clever autoflush detail

by talexb (Chancellor) on Jun 20, 2007 at 19:45 UTC

Re^5: Clever autoflush detail

by Joost (Canon) on Jun 20, 2007 at 19:56 UTC

Re: Clever autoflush detail
by Joost (Canon) on Jun 20, 2007 at 18:44 UTC

Since pre tags have predictable formatting you'll see the content "right away". Adding some other tags around each line, like <br> or <p> will probably work too.

"What should it profit a man, if he should win a flame war, yet lose his cool?"

[reply]
[d/l]
[select]

Re: Clever autoflush detail
by blahblah (Friar) on Jun 20, 2007 at 19:09 UTC

#!/usr/bin/perl -w

$|++;

{
    print "Content-Type:text/html\n\n";

    print "Autoflush is currently ".($|?"on":"off").".\n";
    for ( 1..5 ) {

      print "Value is $_.\n";

      sleep 1;
    }
    print "Autoflush is currently ".($|?"on":"off").".\n";
}
[download]

[reply]
[d/l]

Re^2: Clever autoflush detail

by Joost (Canon) on Jun 20, 2007 at 19:30 UTC

"What should it profit a man, if he should win a flame war, yet lose his cool?"

[reply]

Re: Clever autoflush detail
by eric256 (Parson) on Jun 20, 2007 at 19:52 UTC

Changing just the content type in your first makes it work as well with no pre tags.

In fact if i try your sh script with text/plain it waits and shows them all together, but if i change it to text/html it displays them immediatly. So in both cases it only buffers/caches text/plain for me and text/html works. I would guess then that the problem lies either with the webserver or the web browser.

___________
Eric Hodges

[reply]

Re^2: Clever autoflush detail

by talexb (Chancellor) on Jun 20, 2007 at 20:46 UTC

Well, I tried three of the combinations of (using CGI|not using CGI) and content type (text/html|text/plain):

	CGI	no CGI
text/html	Worked fine using `PRE` tags.	(not tried)
text/plain	Did not work -- buffered all output!!	Worked fine.

I had a look at the CGI source code, and there didn't seem to be anything that talked about ignoring $|, but I will admit I only had a quick look at this Serious Collection of Magic.

In any case, I have a workaround that works (no CGI, text/plain content type) and I'll communicate that back to my client.

But if someone can find out if this deep magic really is happening in CGI, I'd be fascinated to hear about it.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

[reply]
[d/l]
[select]

[OT] Re: Clever autoflush detail ... or ... Autoflush and web browsers
by benizi (Hermit) on Jun 25, 2007 at 02:56 UTC

Re: Update 2. The point of retitling the node is, from my understanding, to make it easier for people searching later on. Even if the discussion has concluded, if I were looking for information on autoflush, the new title would be more useful. If I'm in a situation where the browser might be the problem, the 'browser' in the title can clue me in that the thread is probably useful. And, if my problem has nothing to do with browsers, it's even more important; I'd much rather see "Autoflush and web browsers" instead of the rather vague "Clever autoflush detail", so I could then skip that Super Search result.

[reply]


Don't ask to ask, just ask
	PerlMonks