toadi has asked for the wisdom of the Perl Monks concerning the following question:

This is difficult to formulate. But I have a nice machine besides my (ultra powerful) PIII-256MB I have a p133-64Mb
I'm trying to save memory on this machine and in my hunt I stumbled on a memory hog. My logcoloriser(1.9Mb), I run two of them.

In my inittab I have for tty8 and tty9:
tail -f /var/log/messages | /usr/local/bin/logcolorise
On tty9 I have a network log with logcolorise to ...

So my question is : is there a way(or reading material about it) how I can re-program this program to be less memory-consuming.

I'm looking for general tips so I can do this in the future by myself.


Here is the logcolorise program(it's not my own!!!!):

#!/usr/bin/perl # logcolorize.pl 1.0.8 - A PERL script to colorize log viewing # # Copyright: (C) 1999, Michael T. Babcock # <mikebabcock@pobox.com> # http://www.linuxsupportline.com/~pgp/linux/ # # This program is free software; you can redistribute it and/or modi +fy # it under the terms of the GNU General Public License as published +by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. # # For revisional changes, see ChangeLog file # ###################################################################### $black = "\033[30m"; $red = "\033[31m"; $green = "\033[32m"; $yellow = "\033[33m"; $blue = "\033[34m"; $magenta = "\033[35m"; $purple = "\033[35m"; $cyan = "\033[36m"; $white = "\033[37m"; $darkgray = "\033[30m"; # Define global terms: # Define good notables: @word_good=("starting\n", "Freeing", "Detected", "starting.", "accepte +d.\n", "authenticated.\n", "Ready", "active", "reloading", "saved;", +"restarting", "ONLINE\n"); @word_warn=("dangling", "closed.\n", "Assuming", "root", "root\n", "ex +iting\n", "missing", "Ignored", "adminalert:", "deleting", "OFFLINE\n +"); @word_bad=("bad"); @word_note=("LOGIN", "DHCP_OFFER", "optimized", "reset:", "unloaded", +"disconnected", "connect", "Successful", "registered\n"); @line_good=("up", "DHCP_ACK", "Cleaned", "Initializing", "Starting", " +success", "successfully", "alive", "found", "ONLINE\n"); @line_warn=("warning:", "WARNING:", "invalid", "obsolete", "bad", "Pas +sword", "detected", "timeout", "timeout:", "attackalert:", "wrong", " +Lame", "FAILED", "failing", "unknown", "obsolete", "stopped.\n", "ter +minating.", "disabled\n", "disabled", "Lost"); @line_bad=("DENY", "lost", "shutting", "dead", "DHCP_NAK", "failure;", + "Unable", "inactive", "terminating", "refused", "rejected", "down", +"OFFLINE\n", "error\n", "ERROR\n", "ERROR:", "error", "ERROR", "error +:", "failed:"); # Define specifics: @daemons=("named"); # Now define what we want to use when: $col_good = $green; $col_warn = $yellow; $col_bad = $red; $col_note = $purple; $col_norm = "\033[00m"; $col_background = "\033[07m"; $col_brighten = "\033[01m"; $col_underline = "\033[04m"; $col_blink = "\033[05m"; # Resets to normal colours and moves to the left one column ... $col_default = "$col_norm$white"; print "\n$col_norm$cyan"; print "logcolorize.pl 1.0.8 by$col_brighten$cyan Michael T. Babcock$co +l_norm$cyan <$col_brighten$green"; print "mikebabcock\@pobox.com$col_norm$cyan"; print ">\n"; $datespace=0; mainloop: while (<>) { $thisline = $_; $timestamp = substr($_,0,15); s/................//; @rec = split (/ /, $_); $output="$col_brighten$cyan$timestamp"; # date & time $output.=" $col_brighten$blue$rec[0]"; # hostname if ($rec[1] eq "last") { $output.="$col_norm$green last message repeated "; $output.="$col_brighten$rec[4]$col_norm$green times\n"; print "$output$col_default"; next mainloop; } # Extract PID if present if ($rec[1] =~ /\[(\d+)\]\:/) { my($pid) = $1; $rec[1]=~s/\[$1\]\:// ; $output .= "$col_norm$green $rec[1]" . "$col_brighten$green\["; $output .= "$col_brighten$white$pid" . "$col_brighten$green\]: "; } else { $output .= "$col_norm$green $rec[1] "; } $restcolor="$col_norm$cyan"; $restoftheline=""; for ($therest=(2); $therest<=$#rec; $therest++) { $highlight=0; # Check for keywords to highlight for ($i=0; $i<=$#word_good; $i++) { if ($word_good[$i] eq $rec[$therest]) { $restoftheline.="$col_brighten$col_good"; $highlight=1; +} } for ($i=0; $i<=$#word_warn; $i++) { if ($word_warn[$i] eq $rec[$therest]) { $restoftheline.="$col_brighten$col_warn"; $highlight=1; +} } for ($i=0; $i<=$#word_bad; $i++) { if ($word_bad[$i] eq $rec[$therest]) { $restoftheline.="$col_brighten$col_bad"; $highlight=1; } + } for ($i=0; $i<=$#word_note; $i++) { if ($word_note[$i] eq $rec[$therest]) { $restoftheline.="$col_brighten$col_note"; $highlight=1; +} } # Watch for words that indicate entire lines should be highlig +hted for ($i=0; $i<=$#line_good; $i++) { if ($line_good[$i] eq $rec[$therest]) { $restcolor="$col_norm$col_good"; $restoftheline.="$col_brighten$col_good"; $highlight=1; +} } for ($i=0; $i<=$#line_warn; $i++) { if ($line_warn[$i] eq $rec[$therest]) { $restcolor="$col_norm$col_warn"; $restoftheline.="$col_brighten$col_warn"; $highlight=1; +} } for ($i=0; $i<=$#line_bad; $i++) { if ($line_bad[$i] eq $rec[$therest]) { $restcolor="$col_norm$col_bad"; $restoftheline.="$col_brighten$col_bad"; $highlight=1; } } $restoftheline.="$rec[$therest] "; if ($highlight == 1) { $restoftheline.=$restcolor; } } $output.="$restcolor$restoftheline"; print "$output$col_default\033[1G"; } exit(0);

I know how to benchmark for speed, but this time I want tot reduce memory. Maybe somebody can write a nice tutorial about it.
I feel in these days where ram is cheap nobody cares anymore about writing programs that doesn't eat all your memory... or am I wrong ????
My opinions may have changed,
but not the fact that I am right

Replies are listed 'Best First'.
Re: Memory question
by isotope (Curate) on Oct 18, 2000 at 22:46 UTC
    For starters, you could rewrite it to use File::Tail instead of using tail externally. That might help a bit. I suspect the entire log file is being buffered somewhere.

    Update: Now that I actually *read* the code, I see some problems... the s/........//; is kinda odd... I'm not sure what the author is trying to accomplish, but that will just delete 8 (probably the first 8) characters from each line (maybe he wanted to eliminate literal ........ -- hard to say, and I've never seen that in my logs). If he's trying to delete 8 characters, a substr would be much easier. In fact, unpack might make this thing work better. The biggest thing I see, though, is that all variables are global, so they'll always be in scope. Especially with the variables in the while loop, if you make them local (with my), they'll be destroyed on every iteration, and that might just save you some memory when there's no new line to process. That script is really ugly 8-(. I might try to rewrite it myself later this week if no one else is interested.

    --isotope
    http://www.skylab.org/~isotope/
      Actually,
      each time there a new entry in the log. logcolorise pops up in top using mb of ram.

      Is it with the array... I don't know.


      --
      My opinions may have changed,
      but not the fact that I am right

(tye)Re: Memory question
by tye (Sage) on Oct 19, 2000 at 00:54 UTC

    I see lots of room to improve the speed (replace most of the arrays with hashes). But this would probably increase the memory usage.

    But if it only pops up in top mem usage when it gets activated, then you are talking about working set size, not total virtual memory usage. Predicting max working set size is much harder (at least for me), so I can't see any changes that would clearly reduce it.

    The only place I see to reduce total virtual memory usage is to not append everything into $output when you are just going to print it out. Unless you have very long lines that match a ton of your keywords, I don't think this will make a large improvement.

            - tye (but my friends call me "Tye")