DouglasDD has asked for the wisdom of the Perl Monks concerning the following question:


Short version: How can I get the full command-line process info from 'top' while retaining the ability to redirect STDERR?

Longer Tale of Woe...

I've been working on a script that does some long-term monitoring of the health of a Linux box on one of our customer sites. (So that when we get a bug report that says "the system was really slow for a while last Tuesday after lunch", we'll have some data to look at).

The following 'top' command generates everything that I want to know about the "health" of the Linux box overall as well as the individual processes that we care to monitor:

top -bcS -n 1 -b batch mode (non-interactive) -n <num> iteration count -S cumulative CPU for dead children -c display full command line

Given,

lll.pl: sub longestLineLength { my $longestLen = undef; for my $l ( @_ ) { if ( length($l) > $longestLen ) { $longestLen = length($l); } } return $longestLen; } test-01.pl: #!/usr/bin/perl require "lll.pl"; open (TOP, "/usr/bin/top -bcS -n 1 |") || die "open pipe from top failed $!"; print longestLineLength( <TOP> ) . "\n"; $ ./test-01.pl 80
...Hmm, it looks like `top` is trying to outsmart me because it knows that my terminal is only 80 characters wide (truncating the full command line that I requested with (-c)).

Alright, I can fool it:

Externally:

$ COLUMNS=1000 ./test-01.pl 552

OR Internally test-02.pl:

open (TOP, "COLUMNS=1000 /usr/bin/top -bcS -n 1 |") OR open (TOP, "/usr/bin/env -i COLUMNS=1000 /usr/bin/top -bcS -n 1 |" +)

And all is well:

$ ./test-02.pl 552

BUT as soon as I redirect STDERR, it fails....

$ ./test-02.pl 2>err 81
...once again 'top' is truncating all my command lines!

How can redirecting STDERR possibly relate to terminal width? I don't know, but it behaves the same whether the STDERR re-direct is externally (in the shell as above), or internally (in the perl script below):

test-03.pl: open (TOP, "/usr/bin/env -i COLUMNS=1000 /usr/bin/top -bcS -n 1 2> +err.txt |") $ ./test-03.pl 81
...It doesn't seem to matter whether the destination of STDERR is a real file or /dev/null, the behavior is the same: truncated lines

so next I tried some funky stuff that I learned from 'perlopentut'...

test-04.pl:

#!/usr/bin/perl require "lll.pl"; my $outFileName = shift @ARGV; my $errFileName = $outFileName . ".err"; open( STDOUT, ">> $outFileName" ) || die "failed to redirect STDOUT to '$outFileName': $!"; open( STDERR, ">> $errFileName" ) || die "failed to redirect STDERR to '$errFileName': $!"; open (TOP, "/usr/bin/env -i COLUMNS=1000 /usr/bin/top -bcS -n 1 |" +) || die "open pipe from top failed $!"; print longestLineLength( <TOP> ) . "\n"; $ ./test-04.pl output $ cat output 81
...same result: any attempt at capturing STDERR results in trimmed lines! It doesn't seem to matter whether I use open() or IO::File, the result is exactly the same.

By definition this script is a long-running process, I NEED to capture STDERR somehow, so that I can watch for the error conditions, 'warn's, etc.

I'm at the end of my rope, my next step is to re-write it and replace all occurances of 'warn', 'die', etc. with my own versions that write to a private file-handle (not STDERR). But in addition to being nasty, that's a problem because I'll be losing any STDERR output that is emitted by child processes ('top', etc)).


Relevant Details

$ perl -v This is perl, v5.6.1 built for i386-linux $ type -a top top is /usr/bin/top $ rpm -qf /usr/bin/top procps-2.0.7-11 $ ls -l /bin/sh ... /bin/sh -> bash $ rpm -qf /bin/bash /bin/sh bash-2.05-8 bash-2.05-8 $ uname -a Linux ahost 2.4.9-e.40enterprise #1 SMP Thu Apr 8 16:43:01 EDT 2004 i6 +86 unknown (actually it's RedHat Enterprise Linux 2.1, w/ recent patches)

Apology

Pardon me for asking what might be a Linux question, but I'm hoping that the perl gurus will know of a away that I can work-around it with perl!

Replies are listed 'Best First'.
Re: top and STDERR hell...
by Roy Johnson (Monsignor) on Jun 07, 2005 at 23:52 UTC
    It sounds like top checks whether its output is going to a TTY or not. If not, it ignores TTY settings. You could possibly fool it with a pseudo-TTY.

    Caution: Contents may have been coded under pressure.
      Interesting. An Expect-like approach had not occurred to me, but it sounds like it would be a good way to attack a program like top that is *usually* interactive (though I was hoping to side-step all the terminal-ish issues w/ the -b flag of top.
      <sigh>

      Thanks,
      ./ddd

Re: top and STDERR hell...
by ikegami (Patriarch) on Jun 07, 2005 at 21:45 UTC

    I don't know the answer to your question, but I noticed a bad practice. When including Perl source code that's not in a package, you should use do "filename.pl", not require "filename.pl", because it will fail in this scenerario:

    package AAA; require "111.pl"; longestLineLength(@a); # Works. package BBB; require "111.pl"; # Does nothing. Already required. longestLineLength(@a); # ERROR. -vs- package AAA; do "111.pl"; # Creates &AAA::longestLineLength longestLineLength(@a); # Works. package BBB; do "111.pl"; # Creates &BBB::longestLineLength longestLineLength(@a); # Works.

    Of course, it wouldn't hurt to put the extra line to make it into a module that you could require/use.

      Good point, the require is something that I used just for the demo app. (I haven't put a require in my *real* code since Perl4 days ;-)
        (Sorry, wasn't logged in)
Re: top and STDERR hell...
by tlm (Prior) on Jun 08, 2005 at 01:47 UTC

    FWIW, I get good results with backticks. For example:

    % perl -MList::Util=max -lwe ' print max map length, `COLUMNS=1000 top -bcS -n 1 2>err.txt`' 208
    The output is 81 if I omit the "COLUMNS=1000" bit.

    My system is Debian Linux, and

    % top --version top (procps version 2.0.7)

    the lowliest monk

      Damn, backticks didn't occur to me. (somehow I've gotten in the habit of always using open(H, "app |")). I'll try this out as soon as I get back in the office tomorrow - will let you know.

      Thanks!
      ./ddd

        Actually, it's not a backtick thing; that was just the first thing I tried; when I run your code I also get the desired results. Specifically:

        # lll.pl sub longestLineLength { my $longestLen = undef; for my $l ( @_ ) { if ( length($l) > $longestLen ) { $longestLen = length($l); } } return $longestLen; } 1; # propitiatory offering for require __END__ # test03.pl use strict; use warnings; require "lll.pl"; open (TOP, "COLUMNS=1000 /usr/bin/top -bcS -n 1 2>err.txt|") || die "open pipe from top failed $!"; print longestLineLength( <TOP> ) . "\n"; __END__ % perl test03.pl 208

        the lowliest monk

        Nope. Same behavior with backticks.
        Darn.
Re: top and STDERR hell...
by rdm (Hermit) on Jun 08, 2005 at 03:31 UTC
    In a more general case, and one that is usable for other sites, as well, consider implementing a full historical monitoring system - Cricket is a good option.

    And then implement pacct for the process history.

    -Reality might not get out of Beta today. (O.Timas, "Bot")
      Cricket looks great! Once I get it up and running on our build box, I'll see about adding it to our customer installs!

      Many Thanks!
      ./ddd

Re: top and STDERR hell...
by crusty_collins (Friar) on Jun 07, 2005 at 22:48 UTC
    I ran across this in the Perl cook book .
    Every program starts out with three global filehandles already opened: STDIN, STDOUT, and STDERR. STDIN (standard input) is the default source of input, STDOUT (standard output) is the default destination for output, and STDERR (standard error) is the default place to send warnings and errors. For interactive programs, STDIN is the keyboard, STDOUT and STDERR are the screen:
    while (<STDIN>) { # reads from STDIN unless (/\d/) { warn "No digit found.\n"; # writes to STDERR } print "Read: ", $_; # writes to STDOUT } END { close(STDOUT) or die "couldn't close STDOUT: $!" + }
Re: top and STDERR hell...
by crusty_collins (Friar) on Jun 07, 2005 at 22:23 UTC
    Not sure but I get the same results in KSH running RedHat AS 2.1. So, I am thinking it is a redirection issue?
Re: top and STDERR hell...
by zentara (Cardinal) on Jun 08, 2005 at 12:17 UTC
    Try IPC::Open3:
    #!/usr/bin/perl use warnings; use strict; use IPC::Open3; use IO::Select; my $pid = $$; my $pid1 = open3(0, \*READ,\*ERROR,"top -d 1 -b -p $pid "); #if \*ERROR is false, STDERR is sent to STDOUT my $sel = new IO::Select(); $sel->add(\*READ); $sel->add(\*ERROR); while(1){ foreach my $h ($sel->can_read){ my $buf = ''; if ($h eq \*ERROR){ sysread(ERROR,$buf,4096); if($buf){print "ERROR-> $buf\n"} }else{ sysread(READ,$buf,4096); if($buf){print "$buf\n"} } } }

    I'm not really a human, but I play one on earth. flash japh
      Nope, still 80 cols (even after adding $ENV{'COLUMNS'} = 1000;).

      Neat tho', Both IPC::Open3, and IO::Select are new to me.

      ./ddd

        If you are looking to fool the process into different columns, maybe you need to use IO::Pty. I havn't been able to figure it out myself, but there is something in "clone_winsize_from(\*FH)".

        I have this example in my collection, and I can't remember where I got it...probably google.groups or here. Anyway, it may have a few hints for you.

        #!/usr/bin/perl -w # Description: Fool a process into # thinking that STDOUT is a terminal, when in fact # it may be a file or a pipe. This can be useful # with programs like ps and w on linux... which # will trunc their output to the width of the # terminal, and, if they cannot detect the terminal # width, use a default 80 columns. Wouldn't it be # nice to say "ps -aux | grep etcshadow", and get # output that looks like when you just say "ps # -aux"? Well, that's the idea. #try ./pseudotty "xterm -e top" #or ./pseudotty top use warnings; use strict; use IO::Pty; die "usage: ptyexec command [args]\n" unless @ARGV; my $pty = IO::Pty->new; my $slave = $pty->slave; open TTY,"/dev/tty" or die "not connected to a terminal\n"; $pty->clone_winsize_from(\*TTY); close TTY; my $pid = fork(); die "bad fork: $!\n" unless defined $pid; if (!$pid) { # $slave->close(); open STDOUT,">&=".$pty->fileno() or die $!; exec @ARGV; }else{ $pty->close(); while (defined (my $line = <$slave>)) { print $line; } } #cleanup pty for next run $pty->close();

        I'm not really a human, but I play one on earth. flash japh