shockers has asked for the wisdom of the Perl Monks concerning the following question:

I started with the second example in the Watching_Logs example from the POE cookbook and wanted to make it a little more flexible.

I wanted to have a config file specify what logfiles to look at and what filter to use. I used __DATA__ in my code below for purposes of demonstration.

You can run the script and append anything to file "mylogfile" in the current directory to see it processed by the script.

I managed to do it (and was pleasantly surprised 'eval "\&$filter_name"' worked), but wanted to expand things some more.

Here's my questions:

1. With POE::Wheel::Followtail, how can I pass in the name of the logfile to my "filtering" routine? Or for that matter, how do I pass anything into my filtering routine?

2. Now assume my __DATA__ contents were in a config file. How can I re-read the contents whenever the config file changed? One suggestion was to use SIGHUP, but I don't know where to begin on that. If it involved a fork & exec, isn't POE supposed to be simplifying things to avoid fork & exec? Anyway, one way I CAN do it is a little clumsy -- I can define a "configfile.log" log file that POE looks at. When someone edits the config file, afterwards they append the date/time to the configfile.log and then this script can re-read it. Any cleaner way of doing it?

3. So now I can re-read the contents of the config file after it's been changed, but how do I get the script to use it? I assume I need to stop things and start up again. I see there is a "$poe_kernel_run->()". Is there a "$poe_kernel_stop->()"? I doubt it's that simple, but my guess is I should somehow stop the poe kernel, delete the entire %heap hash, re-define the %heap hash (begin_watchers routine), and then restart $poe_kernel_run().

Could someone explain what I need to do?

In my real application, I want to be looking at a lot of logfiles and to run all the time. I've got a cron running every 5 minutes to restart the script if it dies for some reason. As a side-effect, it would re-read the config file. But I'd rather make things a little cleaner (and more timely) if the application can re-read the config file when the config file changed (save having to kill the task and wait for the cron to start up again).

Thanks.

#!/usr/bin/perl # $Id$ # http://poe.perl.org/?POE_Cookbook/Watching_Logs use warnings; use strict; use POE qw( Wheel::FollowTail ); # Read list of things to watch and filter to use # my %logs_to_watch = | fileidx fullpath # { | -------- -------------------- # cron =>"/var/log/cron", | file0000 => '/var/log/cron' # mail =>"/var/log/maillog", | file0001 => '/var/log/maillog' # ppp =>"/var/log/ppp.log", | file0002 => '/var/log/ppp.log' # httpd=>"/var/log/httpd.log", | file0003 => '/var/log/httpd.log' # msg =>"/var/log/message", | file0004 => '/var/log/message' # myfilter=>"mylogfile1", | file0005 => 'mylogfile1' # }; | my %logs_to_watch; # new stuff my %filter_lookup; # new stuff my $fnum = 0; # new stuff while (<DATA>) # new stuff { # new stuff chomp; # new stuff next if ! /\S/ || /^#/; # new stuff my $fileindex = sprintf "file%04d", $fnum++; # new stuff my ($fullpath, $filter) = (split ' ')[0..1]; # new stuff print "$fileindex : $fullpath : $filter\n"; # new stuff # new stuff $logs_to_watch{$fileindex} = $fullpath; # new stuff $filter_lookup{$fileindex} = $filter; # new stuff } # new stuff # Create hash to replace individual entries in POE::Session->create my %record_hash; # new stuff # new stuff foreach ( keys %logs_to_watch ) # new stuff { # new stuff my $key = $_ . "_record"; # new stuff my $data = '\&' . $filter_lookup{$_} . "_got_record"; # new stuff # new stuff $record_hash{$key} = eval($data); # new stuff } # new stuff # Start a session to watch the logs. POE::Session->create ( inline_states => { _start => \&begin_watchers, # Handle records from each log differently. # cron_record => \&cron_got_record, # old stuff # mail_record => \&mail_got_record, # old stuff # ppp_record => \&ppp_got_record, # old stuff # httpd_record => \&httpd_got_record, # old stuff # msg_record => \&msg_got_record, # old stuff %record_hash, # new stuff # Handle log resets and errors the same way for each file. log_reset => \&generic_log_reset, log_error => \&generic_log_error, } ); sub begin_watchers { my $heap = $_[HEAP]; while ( my ( $service, $log_file ) = each %logs_to_watch ) { my $log_watcher = POE::Wheel::FollowTail->new ( Filename => $log_file, InputEvent => $service . "_record", ResetEvent => "log_reset", ErrorEvent => "log_error", ); $heap->{services}->{ $log_watcher->ID } = $service; $heap->{watchers}->{ $log_watcher->ID } = $log_watcher; } } # Handle log resets the same way for each file. Simply recognize that # the log file was reset. sub generic_log_reset { my ( $heap, $wheel_id ) = @_[ HEAP, ARG0 ]; my $service = $heap->{services}->{$wheel_id}; print "--- $service log reset at ", scalar(gmtime), " GMT\n"; } # Handle log errors the same way for each file. Recognize that an # error occurred while watching the file, and shut the watcher down. # If this were a real log watcher, it would periodically try to resume # watching the log file. sub generic_log_error { my ( $heap, $operation, $errno, $error_string, $wheel_id ) = @_[ HEAP, ARG0, ARG1, ARG2, ARG3 ]; my $service = $heap->{services}->{$wheel_id}; print "--- $service log $operation error $errno: $error_string\n"; print "--- Shutting down $service log watcher.\n"; delete $heap->{services}->{$wheel_id}; delete $heap->{watchers}->{$wheel_id}; } # Filters for each log file ( shortened to save space ) sub cron_got_record { my $log_record = $_[ARG0]; print "cron_got_record: "; print "$log_record\n"; return; } sub mail_got_record { my $log_record = $_[ARG0]; print "mail_got_record: "; print "$log_record\n"; return; } sub ppp_got_record { my $log_record = $_[ARG0]; print "ppp_got_record: "; print "$log_record\n"; return; } sub httpd_got_record { my $log_record = $_[ARG0]; print "http_got_record: "; print "$log_record\n"; return; } sub msg_got_record { my $log_record = $_[ARG0]; print "msg_got_record: "; print "$log_record\n"; return; } sub myfilter_got_record { my $log_record = $_[ARG0]; print "myfilter_got_record: "; print "$log_record\n"; return; } sub reread_got_record { my $log_record = $_[ARG0]; print "reread_got_record: "; # &reread_configfile (\%logs_to_watch, \%filter_lookup); # print "$log_record\n"; return; } # Run the watchers. The run() method will return after all the # watchers end. $poe_kernel->run(); __DATA__ # fullpath filter # ------------------------ --------- /var/log/cron cron /var/log/maillog mail /var/log/ppp.log ppp /var/log/httpd-access.log httpd /var/log/messages msg mylogfile myfilter configfile.log reread

Replies are listed 'Best First'.
Re: Adding flexibility to Watching_Logs example in POE Cookbook
by Ultra (Hermit) on Nov 01, 2005 at 08:31 UTC

    There may be other ways, here is mine (unified diff, you need to patch your program):

    Basically you need to receive an event when config file changes. You can setup an alarm to check this periodically, or any other means to be immediately notified.

    Then, re-read your configuration file (you need to get rid of _DATA_ and use a procedure that returns you the hash), get the new files, delete your old FollowTail wheels, and create brand-new ones using begin_watchers.

    I get advantage of the fact that you use %logs_to_watch like a "global", but I'd advise you to store all your "globals" to the HEAP.

    Oh and a final note: the program doesn't change the log files, just shows you how to renew your wheels. And it isn't tested ;-)

    Dodge This!
      Thanks. Your advice helped some. I put the stuff in __DATA__ to a file called "watchlist". I put the code to read the watchlist into a subroutine. When the program starts or I detect the watchlist changed, I call the subroutine to read the watchlist. The method I used to detect a change to the watchlist was to add "configfile.log" into the watchlist. When I append date/time to "configfile.log", it calls the routine to read the watchlist.

      In your code, I see how you delete the heap's 'services' and 'watchers' in alarm_reset. Should I call that in my routine that re-reads the watchlist? When the attached code runs, alarm_reset gets run every 10 seconds, so it seems the 'services' and 'watchers' are not getting cleared out.

      I confirmed I can reread the watchlist, but it seems to me that for the "new" watchlist to take effect (wiping out any prior watchlist entries that are no longer in the new watchlist), I feel the original $poe_kernel->run() should somehow be stopped, the original POE::Session->create needs to be destroyed, another POE::Session->create should be run, and then a new $poe_kernel->run() be started.

      Is this a false assumption? Or is your alarm_reset deleting entries in the heap and I should somehow dynamically enter replacement entries without stopping the poe_kernel and re-creating the poe session?

        Interesting approach, tailing configfile.log

        The example with the alarm was just an example trigger, to show you how you can re-read the configuration. You can get rid of it now, as you have a trigger (configfile.log)

        Say you have four files to watch, namely file A, ... file D. You create for each one of them a Wheel, that you store in $_HEAP->{'watchers'}. I'd suggest you keep the wheels like: $_HEAP->{'watchers'}->{'A'} .. $_HEAP->{'watchers'}->{'D'} etc.

        This way, when you send the event that the configuration file has changed, a subroutine checks the new file names to tail. Say you need to watch now file A, file B, file E and file Z. The subroutine will keep the wheels stored in $_[HEAP]->{'watchers'}->{'A'} and $_[HEAP]->{'watchers'}->{'B'}, deletes the wheels watching files C and D, and creates new wheels for files E and Z

        The whole point is that you are keeping the Wheels alive, by having a reference to them in the $_[HEAP]. As soon as you delete a reference to one wheel from $_[HEAP], $poe_kernel will do the clean-up for you (in case there are no more events for it to be delivered or there is no other reference to them).

        There are other methods too to keep sessions alive, Here's another example

        In short, you don't need to restart the $poe_kernel. It's main purpose is to live forever ;-) But you may "restart" wheels by deleting, recreating them as needed.

        Dodge This!