Wen has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks, I am a Perl newbie and hope you can point me in the right direction. I am writing script, which will look at a certain records in system log files. The problem is that it may need to check the records going up to few days back. By the time of running the check, log files may have already been rotated and archived as bz2 in the same /var/log directory.

What is the most efficient way to "go back in time" and search in these rotated files? Since logs are rotated by size, not date, the script does not know through how many files back it has to look, until it looks inside the file.

Thank you for advice

Replies are listed 'Best First'.
Re: How to read from rotated log file?
by JavaFan (Canon) on Jan 29, 2012 at 21:16 UTC
    I assume your system numbers log files, and that compressed log files end in .gz.

    Let's assume your base file is called "my-app". Then it's time to use magic open! Untested: 

    my @files = </var/log/my-app*>;
/\.gz/ and $_ = "zcat $_ |" for @files;
local @ARGV = @files;
while (<>) {
    ... do something with record ...
}

    [download]
    This will read uncompressed files in the usual way, and read from a pipe (with zcat on the other end) for compressed files.
[reply]
[d/l]
Re: How to read from rotated log file?
by johngg (Canon) on Jan 29, 2012 at 23:47 UTC

    If you know the date you want to go back to then you should be able to convert that date into epoch seconds (see Time::Local) and compare with the time last modified (see stat) of your compressed log files to see which one to examine.

    If you are using 5.12 or later you should be able to use the built-in IO::Uncompress::Bunzip2 module to read the file. 5.10 or earlier and you will have to use the bunzip2 executable the way JavaFan has shown for gunzip.

    Cheers,

    JohnGG

[reply]
      Thank you, JohnGG. It is a great idea to look at the file creation dates and I used the piece of code by JavaFan with slight modifications. Thanks for your help, guys!
[reply]

Back to Seekers of Perl Wisdom