in reply to Suffering with buffering - how to wake 'less'

Try this: 

#! perl -sw
use strict;
use threads;
use Thread::Queue;

my $Q = new Thread::Queue;

async{
    $Q->enqueue( $_ ) while <>;
    $Q->enqueue( undef );
}->detach;
sleep 1;

print while $_ = $Q->dequeue;

[download]

Name it buf(.pl) and use it like this: 

perl -E"$_%1000 or say 'Here'. $_ for 1..1e6" | buf | less

[download]
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"I'd rather go naked than blow up my ass"

Replies are listed 'Best First'.
Re^2: Suffering with buffering - how to wake 'less'
by benizi (Hermit) on Jan 13, 2010 at 19:43 UTC

    Wow. Cool.

    I was worried it wouldn't work, since your test wasn't testing the same thing as mine. (The warnings were a side channel indication of the fact that 'less' was stopping 'perl'). But, it seems to do the trick. Thanks a bundle.

    ...Except now I'm confused. How does it work? What is buf(.pl) doing that prevents 'less' from stopping it? or that forces 'less' to keep reading? I'm kind of back to my initial question from another angle: what is 'less' doing to 'perl' that indicates to 'perl' that it should pause its output?

[reply]

      Once less has displayed the first screen full, it stops reading the pipe and waits for instructions from the keyboard. The pipe fills and won't accept anymore from the producer, so it blocks on the write until the pipe empties. Which it won't until you hit the space bar or similar.

      buf.pl uses a shared queue as a buffer between two threads, The read thread just keeps reading and pushing to the queue. The write thread reads from that queue and writes to the pipe. Because the two are asynchronous, when the write thread blocks because the pipe to less is full, the reader just carries on filling up memory until the producer finishes.

      The downside is that if the producer produces sufficient output, the perl process will eventually run out of memory and die.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        There it is, thanks. I was lost by the disconnect between the two facts: 1. 'less stops after one screen' and 2. 'perl produces more than one screenful before it pauses'.

[reply]

      What is buf(.pl) doing that prevents 'less' from stopping it?

      Nothing. buf's output thread is still blocking at the same spot as the producer would without buf.

      less stops reading when it has enough to display what it needs to display. The producer doesn't know or care when that happens, so it keeps adding to the pipe until it fills up. Trying to add more to the pipe, the producer blocks until less starts emptying it.

      buf reads all that it can as fast as it can. That's likely to be faster than the producer can produce output, so the pipe from the producer never fills, so the producer never blocks. buf keeps the lines it read in a Queue from where the writer will fetch them when it unblocks.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom