Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Suffering with buffering - how to wake 'less'

by benizi (Hermit)
on Jan 12, 2010 at 19:43 UTC ( #817033=perlquestion: print w/replies, xml ) Need Help??

benizi has asked for the wisdom of the Perl Monks concerning the following question:

The only thing that generally annoys me about the 'less' command (other than trying to google it) is that I can't figure out how to get it to not stop the program on the input end of its pipe. Take the following example:

perl -lwe 'print and /000$/ and warn "HERE$_\n" for 1..1e6' | less

HERE1000 through HERE14000 get printed to STDERR before less stops perl. Unbuffering output (select((select($_),$|=1)[0]) for STDOUT, STDERR) doesn't change anything.

perl isn't the program with which I'm having trouble, but hopefully what works for perl will point me in the right direction.

Replies are listed 'Best First'.
Re: Suffering with buffering - how to wake 'less'
by ikegami (Patriarch) on Jan 12, 2010 at 20:09 UTC

    Printing to a full pipe causes the print to block. less doesn't read from its input until it needs to. I don't see any option to change that behaviour.

      Nuts. I was hoping for something like +G or +F that wouldn't cause less to fully block.

      Falling back to the ol' dosomething >& file &; less file approach.

      Thanks as always ikegami.

        producer > file & less file ; rm file
        doesn't work since less isn't expecting the file to change.
        producer > file ; less file ; rm file
        works, but less will only start up once the producer has ended.
Re: Suffering with buffering - how to wake 'less'
by BrowserUk (Patriarch) on Jan 12, 2010 at 22:51 UTC

    Try this:

    #! perl -sw use strict; use threads; use Thread::Queue; my $Q = new Thread::Queue; async{ $Q->enqueue( $_ ) while <>; $Q->enqueue( undef ); }->detach; sleep 1; print while $_ = $Q->dequeue;

    Name it buf(.pl) and use it like this:

    perl -E"$_%1000 or say 'Here'. $_ for 1..1e6" | buf | less

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      Wow. Cool.

      I was worried it wouldn't work, since your test wasn't testing the same thing as mine. (The warnings were a side channel indication of the fact that 'less' was stopping 'perl'). But, it seems to do the trick. Thanks a bundle.

      ...Except now I'm confused. How does it work? What is buf(.pl) doing that prevents 'less' from stopping it? or that forces 'less' to keep reading? I'm kind of back to my initial question from another angle: what is 'less' doing to 'perl' that indicates to 'perl' that it should pause its output?

        Once less has displayed the first screen full, it stops reading the pipe and waits for instructions from the keyboard. The pipe fills and won't accept anymore from the producer, so it blocks on the write until the pipe empties. Which it won't until you hit the space bar or similar.

        buf.pl uses a shared queue as a buffer between two threads, The read thread just keeps reading and pushing to the queue. The write thread reads from that queue and writes to the pipe. Because the two are asynchronous, when the write thread blocks because the pipe to less is full, the reader just carries on filling up memory until the producer finishes.

        The downside is that if the producer produces sufficient output, the perl process will eventually run out of memory and die.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        What is buf(.pl) doing that prevents 'less' from stopping it?

        Nothing. buf's output thread is still blocking at the same spot as the producer would without buf.

        less stops reading when it has enough to display what it needs to display. The producer doesn't know or care when that happens, so it keeps adding to the pipe until it fills up. Trying to add more to the pipe, the producer blocks until less starts emptying it.

        buf reads all that it can as fast as it can. That's likely to be faster than the producer can produce output, so the pipe from the producer never fills, so the producer never blocks. buf keeps the lines it read in a Queue from where the writer will fetch them when it unblocks.

Re: Suffering with buffering - how to wake 'less'
by ahmad (Hermit) on Jan 12, 2010 at 22:21 UTC

    You could try using another command, like:

    perl -lwe 'print and /000$/ and warn "HERE$_\n" for 1..1e6' | tail -f

      That omits the first 999,990 lines of that sample script. But, as used in my reply to ikegami, it's close to what I'm settling on. dosomething > file &; tail -f file Even though I really just want some 'read the rest of the input' command in 'less'.

        producer > file & tail -n +1 -f file | less ; rm file

        Nah, no good, tail never exits, so less freezes trying to get the end of the file.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://817033]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (5)
As of 2022-08-14 08:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?