r has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I have a really buggy server that crashes a couple of times a day; notable, the process itself never dies. I analyzed the error logfile and found out that at least two regular expressions precede a hangup of the server. Now comes the problem, the servers logrotation does not work in the intended way. In fact, once a day it rotates the error logfile from errorlog to i.e. errorlog.04072002 but always at a different time. Not enough, at times it does not create a new errorlog file but continues to write in the renamed logfile! So I have to take care if there is an errorlog knowing that that I have to reopen the filehandle if the logfile rotates correctly. My script is running permanently in the background checking continuously (tail -f alike).

And here it is, my current version of the checkscript, that stops and afterwards starts the server if the regex is found in the errorlog. Somehow I still have problems with it because after one restart a lot of them follow. I am pretty sure that it could be done better so that I can guarantee that downtimes are minimized and, of course, are not provoked by the check script itself ;-)
Awaiting your (tested) solutions with great thankfulness,

#!/usr/bin/perl

use strict;

my $debug = 1;

my $serverinst  = "/opt/myserver";
my $CLIENTLOG   = "$serverinst/logs/errorlog";
my $logfile     = "/opt/mon/check.log";
my $pidfile     ="/opt/mon/check.pid";

my $errpatt1     = "Internal Error";
my $errpatt2     = "Error 0x0c543 con";
my $polltime     = 10; # poll for new file every x seconds

open (PIDFILE,">$pidfile") or die "Couldn't open $pidfile: $!\n";
print PIDFILE "$$";
close PIDFILE;

open (CL,"$CLIENTLOG") or die "Couldn't open $CLIENTLOG: $!\n";
my $inode = (stat $CLIENTLOG)[1];

### read to end of file
seek (CL, 0, 2);

logit(" ### Started $0 with PID $$ ###\n");

LINE:
while(1) {
  logit ("reading") if $debug;
  while (<CL>) {
    logit ("read one line");
    if (/$errpatt1/ or /$errpatt2/) {
      wsrestart();
    }
  }
  if ( -f $CLIENTLOG) {
    if (get_inode($CLIENTLOG) != $inode) {
      logit ("inode changed to [$inode]");
      unless (open (CL, $CLIENTLOG)) {
        logit ("ERROR opening $CLIENTLOG: $!");
        die "ERROR opening $CLIENTLOG: $!\n";
      }
      $inode = get_inode($CLIENTLOG);
      next LINE;
    }
  }
  logit ("sleeping") if $debug;
  sleep ($polltime);
  seek (CL, 0, 1)
}

sub logit {
  my $message = shift;
  open (LOG, ">>$logfile") or die "Couldn't open $logfile: $!\n";
  print LOG localtime(time())."\t$message\n";
  close LOG;
}

sub get_inode {
  return (stat $_[0])[1];
}

sub wsrestart {
  system("$serverinst/stop");
  logit("Server instance $serverinst stopped");
  sleep(3);
  system("$serverinst/start");
  logit("Server instance $serverinst started");
}

[download]

The waiting ones may be too late,
the impatient could soon have nothing worth to wait for.

-r

Replies are listed 'Best First'.
•Re: File tracking
by merlyn (Sage) on Jul 06, 2002 at 15:09 UTC
    Here's what I'm using to rotate logs on my new shiny website: 
    #!/usr/bin/perl -w
use strict;
$|++;

### Processed automatically at 10:29:34 21-Jun-2002 from              
+     ###
### sbin/log-roller.tmpl edited at 07:58:15 17-May-2002               
+     ###

my $PIDFILE = "/web/stonehenge/var/run/httpd.pid";
my @LOGS = glob("/web/stonehenge/var/log/*_log");

## first, compress any old log files

if (my @OLDLOGS = grep -M > 7, glob("/web/stonehenge/var/log/*[0-9]"))
+ {
  system 'gzip', '-9', @OLDLOGS;
}

## then, get the PID

open PIDFILE, $PIDFILE or die "Cannot open $PIDFILE: $!";
chomp(my $pid = <PIDFILE>);
close PIDFILE;
$pid =~ /\d+/ or die "$pid doesn't look like a pid from $PIDFILE";

## now roll the logs

{
  local *ARGV;
  local $^I = "." . time;
  @ARGV = @LOGS;
  close ARGV while <>;          # quick rename while emptying
}

## finally, let Apache know we've messed around

kill 'USR1', $pid or warn "Cannot send USR1 to $pid: $!";

    [download]
    That last step is critical. There must be some way to let the writer know that the file they have open is now a new file.

    -- Randal L. Schwartz, Perl hacker

[reply]
[d/l]
      Deeply appreciating your advise and sharing of wisdom I have to confess in meekness that my question apparently has not been put in a sufficiently stringent manner.
      To make it clear now, the mentioned server is no Apache, and I am not in the position to change the logfile rotation procedure in this productive system :-(

      I will pose some subquestions to clarify:

      1) Is my implementation of "tail -f" neat enough? I think so.
      2) Assuming that logfile rotation is working correctly, is the reopening of the current logfile done properly assuring no loss of data?
      3) The real whole problem - how to manage the following possibilities:
      a) rotated logfile with altered name may still be the active, no logfile with the original name present -> No problem (see inode comment in source)
      b) at some indefinite time a new logfile with the original name becomes the active one -> Point 2

      I ask for patience and balminess,
      bearing in mind my own weakness.

      -r
[reply]

        Is my implementation of "tail -f" neat enough? I think so.

        I don't know if sleep/poll cycles are the most efficient way of doing things here. You might want to try using select if you can.

        ... is the reopening of the current logfile done properly assuring no loss of data

        I'm not sure.. imagine this scenario:

        1. Your process scans the logs.
        2. Your process sleeps.
        3. While you are sleeping, the server outputs the bad error lines.
        4. Still while you are sleeping, the logs get rotated.
        5. When you wake up, you'll start processing the new file and never see the error lines.

[reply]

      $pid =~ /\d+/ or die "$pid doesn't look like a pid from $PIDFILE";

      Does QWERTY123 look like a pid then? :) 

      $pid =~ /^\d+\z/ or die;

      [download]

      - Yes, I reinvent wheels.
- Spam: Visit eurotraQ.

[reply]
[d/l]
[select]
Re: File tracking
by ehdonhon (Curate) on Jul 06, 2002 at 18:44 UTC

    Not a perl solution, but you might want to take a look at newsyslog as a solution for log rotation. You could even configure it to send a HUP signal to your log monitor whenever it rotates the logs so that your program would know that it is time to re-open the filehandle.

[reply]

Back to Seekers of Perl Wisdom