alager has asked for the wisdom of the Perl Monks concerning the following question:

When I run my script it runs okay, but there are major delays in the output. Each child runs a remote ssh command on the IPs listed in %hosts. The command can optionally take grep arguments. It seem like the last child gets priority in printing to STDOUT. What am I missing? Why don't all three children print nearly evenly and why the delays? The delays are as follows, I can open another window and watch directly on the remote server using tail -f, as the log file grows. But the output of my script is always 3-4 log entries behind. The log gets updated about every 10 seconds, so 40 seconds is a long time. Thanks, Aaron
#!/usr/bin/perl $|=1; %hosts=( "host1" => "192.168.168.32", "host2" => "192.168.168.33", "host3" => "192.168.168.34" ); $file = "/var/log/bluebolt/bluebolt.log"; $tail = "/usr/bin/tail"; $ssh = "/usr/bin/ssh"; $grep = "/bin/grep --color=always"; #die "Usage: $0 <file>\n" unless @ARGV; if ($ARGV[0] eq "--help"){ print "Usage $0 [pattern]\n"; print "$0 will show all log files for App servers\n"; print "$0 [pattern] will grep the log files for the pattern\n" +; exit; } $SIG{CHLD} = 'IGNORE'; $SIG{INT} = sub {done("Received SIGINT")}; sub done { print "@_\n"; exit; } foreach $key (keys %hosts) { if(fork){ #parent process does nothing } else { #child process does all the work #add any grep commands here if ($ARGV[0]) { if ($ARGV[1]) { $command = "$ssh $hosts{$key} $tail - +f $file | $grep $ARGV[0] $ARGV[1]"; }else{ $command = "$ssh $hosts{$key} $tail - +f $file | $grep $ARGV[0]"; } } else { $command = "$ssh $hosts{$key} $tail -f $file"; } open (README, "$command |") or die "Couldn't fork: $!\ +n"; select README; $|=1; while(<README>){ print STDOUT "$hosts{$key}: "; print STDOUT $_; } } } while(1){sleep(5);}#parent process needs to wait for Ctrl-C exit;

Replies are listed 'Best First'.
Re: why the long delays?
by ikegami (Patriarch) on Apr 15, 2010 at 17:23 UTC
    You made sure your output isn't buffered ($|=1;), so that means the it must be one of the other programs that is buffering its output. ssh, tail and/or grep realize they are connected to a pipe and buffer their output as a result. You'll need to convince the offender(s) that they are connected to a terminal via some command line option or by using a pseudo tty (ptty).

    For my versions of tail, grep and ssh, I found that

    • tail -f doesn't buffer its output.
    • grep buffers its output unless it's output is connected to a terminal or --line-buffered is provided.
    • ssh doesn't buffer its output.

    Adding --line-buffered to grep should do the trick.

    Update: Added "For my version" on.

      Adding --line-buffered to grep **DID** do the trick. Thanks for the help.

      The reason I stuck with the shell grep instead of the perl grep was the ease of highlight on match.

      Aaron

Re: why the long delays?
by MidLifeXis (Monsignor) on Apr 15, 2010 at 17:22 UTC

    tail and grep both buffer at the application level, IIRC. Unless you can convince those applications not to buffer their output, this may be the best you can do using the shell.

    Each of these functions, OTOH, is replicated in Perl, either in core (grep) or in a module (tail, ssh).

    My wife is walking for a cure for MS. Please consider supporting her.