wishartz has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to create a log watcher, that watches multiple log files on linux, like /var/log/messages and /var/log/secure and writes any new messages to a file. After the two minute interval it will check again and write to a new file, it will then compare the two files and save any messages that were not there before and check that they are not messages that need to be excluded, from the exclude config files and write them to a third file. This third file will then be displayed in a an xterm console window using less, to notify operators of any outstanding problems that need to be reported. I have been using POE::Wheel::FollowTail to monitor multiple files at any one time. This is what I have so far, but I do not know how to compare the two files and write to a third file to be displayed. There will be an xterm window every two minutes for each separate log file alert, if there are any messages that the operators need to take notice of, which the operators can close, with the 'q' key, because I will be using the less command to display the file. I seem to have reached a dead end with the program. Can anybody help please.
#!/usr/bin/perl use POE qw/Wheel::FollowTail/; use strict; use warnings; our $filename; our $output="output"; open(CONFIG_M, "exempt.messages") || die("Could not open file!"); open(CONFIG_S, "exempt.secure") || die("Could not open file!"); my @exempt_messages=<CONFIG_M>; close(CONFIG_M); my @exempt_secure=<CONFIG_S>; close(CONFIG_S); open(OUTPUT,">>$output") || die("Cannot Open File"); my %logs_to_watch = ( secure => "/var/log/secure", msg => "/var/log/messages", ); # Start a session to watch the logs. POE::Session->create ( inline_states => { _start => \&begin_watchers, # Handle records from each log differently. secure_record => \&secure_got_record, msg_record => \&msg_got_record, # Handle log resets and errors the same way for each file. log_reset => \&generic_log_reset, log_error => \&generic_log_error, } ); sub begin_watchers { my $heap = $_[HEAP]; while ( my ( $service, $log_file ) = each %logs_to_watch ) { my $log_watcher = POE::Wheel::FollowTail->new ( Filename => $log_file, PollInterval => 120, InputEvent => $service . "_record", ResetEvent => "log_reset", ErrorEvent => "log_error", ); $heap->{services}->{ $log_watcher->ID } = $service; $heap->{watchers}->{ $log_watcher->ID } = $log_watcher; } } # Handle log resets the same way for each file. Simply recognize that # the log file was reset. sub generic_log_reset { my ( $heap, $wheel_id ) = @_[ HEAP, ARG0 ]; my $service = $heap->{services}->{$wheel_id}; print "--- $service log reset at ", scalar(gmtime), " GMT\n"; } # Handle log errors the same way for each file. Recognize that an # error occurred while watching the file, and shut the watcher down. # If this were a real log watcher, it would periodically try to resume # watching the log file. sub generic_log_error { my ( $heap, $operation, $errno, $error_string, $wheel_id ) = @_[ HEAP, ARG0, ARG1, ARG2, ARG3 ]; my $service = $heap->{services}->{$wheel_id}; print "--- $service log $operation error $errno: $error_string\n"; print "--- Shutting down $service log watcher.\n"; delete $heap->{services}->{$wheel_id}; delete $heap->{watchers}->{$wheel_id}; } # Display some interesting things from the messages log. sub msg_got_record { my $log_record = $_[ARG0]; print "$log_record\n"; print OUTPUT " $log_record\n"; #system ("xterm less output"); foreach my $ignored (@exempt_messages) { return if $log_record eq $ignored; } } sub secure_got_record { my $log_record = $_[ARG0]; print "$log_record\n"; print OUTPUT " $log_record\n"; #system ("xterm | less output"); foreach my $ignored (@exempt_secure) { #Not sure how to ignore messages in exempt_secure # system ("/bin/grep -v $ignored $log_record # return if $log_record eq $ignored; } } POE::Kernel->run(); exit;

Replies are listed 'Best First'.
Re: Log watcher which outputs alerts in xterm windows
by quester (Vicar) on Jun 23, 2008 at 08:01 UTC

    Actually, my advice would be to get a copy of swatch and start with it. This is probably the most common solution to the problem of watching log files. It is an optional package on most Linux distributions. If it doesn't do everything you want, it is written in Perl, so you can just edit it. Note that it solves several problems besides the ones that you mention, like playing well with logrotate when it truncates the log files.

    Out of the box, it doesn't have the ability to monitor more than one log file at once. You could run a copy of swatch for each file, each with its own output window. Or, you could hack the source of swatch to allow multiple input files. Or, try this: create a named pipe with mkfifo, run one copy of tail -f for each input file, all sending their output to the pipe, and have swatch read the pipe as its input.

    Whatever you do, don't allow your monitoring process to open a new window every two minutes without closing any of the previous ones automatically. If the graveyard shift operator catches the flu and the system is left unattended for many hours, you will have hundreds of copies of both less and xterm running. Depending on the system configuration, you might risk having the system run short of memory and do Bad Things.

      Thanks for your advice about swatch, I'm going to give that a try, but I'm still going to try and make this program work to try and improve my perl skills. I just had an idea of rather than comparing two files, I could compare two arrays. Since the program is going to check every x amount of minutes, I could store one of the arrays as an old array and then 5 minutes later when it checks, I could slurp the burst of messages in a new array, those two arrays could then be compared and any different lines, could be written to a file and then outputted to an xterm window using the less command. I don't really know how to do it though. Does it seem possible?
Re: Log watcher which outputs alerts in xterm windows
by zentara (Cardinal) on Jun 23, 2008 at 13:30 UTC
    Since you want to display results in an xterm, it means you have an X server running, so why not use Tk, Gtk2, Wx, etc. to do this? You could pop up windows with text boxes for the display, the 'q' key control would be easy, and you can setup timers. Nothing against POE, but did you choose it because you found some followtail code you thought you could modify? You can search groups.google.com for "perl tk tail file" for example code.

    I'm not really a human, but I play one on earth CandyGram for Mongo