Reading from a command pipe

DimosTsag has asked for the wisdom of the Perl Monks concerning the following question:

Hello everybody.

I am writing code for a TK GUI, from which I am running a shell command and display its output on the window. I have also included a progressbar. The backbone of my code is shown below

#!/usr/bin/perl

use strict;
use Tk;
use Tk::ROText;
use Tk::ProgressBar;

my ($complete_steps,$steps,$progress,$pid) = (14,0,0,undef);

my $mw = new MainWindow;
my $button = $mw->Button(-text => 'Run', -command => sub{&run})->pack(
+);
my $pb = $mw->ProgressBar(-variable=>\$progress,-width => 20,-blocks=>
+100,-gap => 0)->pack(-fill => 'x', -expand => 1);
my $output = $mw->Scrolled('ROText', -scrollbars => 'osoe', '-wrap','n
+one')->pack(-anchor => 'nw', -fill => 'both', -expand => 1);

MainLoop();

sub run {
    my $cmd = "...";
    my ($flag,$cmd_handle);
    $pid = open ($cmd_handle, "$cmd 2>&1 |") or $flag = $!;
    unless ($flag){
        $mw->fileevent($cmd_handle,'readable', [\&working,$cmd_handle]
+);
    } else{}    
}

sub working {
    my ($cmd_handle) = @_;
    my $redline;

    my $stat = sysread $cmd_handle, $redline, 4096;
    unless ($stat == 0) {
        $output->insert('end', "$redline");$output->yview('end');
        &update_remaining($redline);
    } else {
        $mw->fileevent($cmd_handle ,'readable','');
        $steps = 0;
        $progress = 0;
    }
}

sub update_remaining {
    my ($redline) = @_;
    my @list= ($redline =~ /Now/g);
    $steps += scalar(@list);
    my $temp = 100 * ($steps/$complete_steps);
    $progress = $temp;
}
[download]

The problem is that while everything works ok and nothing blocks, perl seems to accepts the data from the pipe in batches (around 4000 characters at a time). When running the command directly in the shell, the output is gradually shown (line by line). I assume that "somewhere" between the command and the output window there is a buffer that I cannot locate and this messes up the progress bar smoothness.

I have checked that the number of bytes I use in the sysread doesn't play a role in that. So, any help is welcome.

Thanks in advance for your time.

Comment on Reading from a command pipe Download Code

Replies are listed 'Best First'.
Re: Reading from a command pipe by MidLifeXis (Monsignor) on Feb 09, 2012 at 13:14 UTC
I am guessing that it is around 4096 characters at a time (2**12). You are possibly Suffering from Buffering. In the pipe's source program, make sure that you have output buffering turned off. In a Perl program, that would be done with the $\| variable. If the program you are feeding the pipe with does not allow you to turn off buffering, there is not a lot that you can do. --MidLifeXis	[reply]
Re: Reading from a command pipe by kcott (Archbishop) on Feb 09, 2012 at 13:54 UTC
You may want to call the `update()` method to refresh the display of your ROText and/or ProgressBar widgets. The `update()` method can slow things down a bit, so you'll probably want to experiment to find a happy medium between the visual experience and how long the overall process takes to run. `update()` is a standard widget method documented in Tk::Widget. -- Ken	[reply] [d/l] [select]
Re: Reading from a command pipe by DimosTsag (Initiate) on Feb 09, 2012 at 14:17 UTC
Hi. Thank you for your quick responses. I have tried $mw->update() with no success. After reading the links about buffering I tried making the $cmd_handle hot, with select and $\|=1 (didn't work either) and with autoflush (it threw an error). The first comment helped me though to get into thinking. While running the cmd directly in the shell did a line by line output, when trying to pipe it also from the shell, e.g. cmd \| less, it presents the same problems as invoking from the perl script. So the problem must lay in the cmd I call. It is a c compiled program, but I also have the source code, so I will try my luck there now. It is new and strange to me that outputing in STDOUT dumps line by line, but piping to another program is dumped block by block. Thanks again for your help.	[reply]
Re^2: Reading from a command pipe by Eliya (Vicar) on Feb 09, 2012 at 15:25 UTC
It is new and strange to me that outputing in STDOUT dumps line by line, but piping to another program is dumped block by block. This is the normal/default behavior. Quoting from the man page of setvbuf(3) (the C lib function to change the buffering mode): The three types of buffering available are unbuffered, block buffered, and line buffered. When an output stream is unbuffered, information appears on the destination file or terminal as soon as written; when it is block buffered many characters are saved up and written as a block; when it is line buffered characters are saved up until a newline is output or input is read from any stream attached to a terminal device (typically stdin). The function fflush(3) may be used to force the block out early. (See fclose(3).) Normally all files are block buffered. When the first I/O operation occurs on a file, malloc(3) is called, and a buffer is obtained. If a stream refers to a terminal (as stdout normally does) it is line buffered. The standard error stream stderr is always unbuffered by default. (emphasis mine) When you pipe stdout to another program, it is no longer connected to a terminal, hence it is block-buffered by default.	[reply]
Re: Reading from a command pipe by DimosTsag (Initiate) on Feb 09, 2012 at 15:51 UTC
Hi again. I have found a solution to my problem. It is not relevant with perl, since the problem wasn't either as I found out, but to rup this thread up, I feel I must post the solution also. The problem is that when you pipe a command (redirect STDOUT to another command) in unix (and as a consequence also with the open perl command), the default behaviour of outputing lines changes to outputing blocks, for less read callbacks to the system. The solution if you want to keep the line to line dumping is to use the shell command "stdbuf (with -oL in my case) $cmd". Hope this may be of use to someone else in the future. Bye	[reply]