Correlating Log files

bsimonds has asked for the wisdom of the Perl Monks concerning the following question:

I want to look at multiple log files and correlate the messages in each based on x minute intervals. This is in an effort to trouble shoot an issue. My initial thoughts are to create a hash with a descriptive name for a log as the key, and then the value being an array with the path to log file and a place to put a reg ex to match the date. (Unfortunately, the logs to not use the same formatting.) Ex:

$hash{apache_access} = ('/path/to/apache_access.log', 
                         "^\d\d\/\w\w\w\/\d\d\d\d\:\d\d\:\d\d:\d\d");
[download]

What i am looking for is an efficient way to coordinate the error messages from different logs in to one data structure and either print that to the screen or dump it to a file. I am worried about the size of the structure, as it could potentially get very large very fast since I will be looking at approximately 7 log files (including switch and firewall logs) for about a 4 hour period on 1 minute intervals. Any suggestions would be much appreciated.

Comment on Correlating Log files Download Code

Replies are listed 'Best First'.
Re: Correlating Log files by tilly (Archbishop) on Jan 14, 2009 at 20:59 UTC
The simplest solution is to define a table in a database that will hold a single record from any of the logs. Then write 7 jobs that parse each log and puts it in the database. Add an index on the log time. Then you can get your 1 minute intervals by queries against that database table. When you're done, drop the table.	[reply]
Re: Correlating Log files by jethro (Monsignor) on Jan 14, 2009 at 21:55 UTC
More work but without the need for a database would be a merge sort. Just open all files simultaneously and in each step check which of the 7 files has the oldest date at the current file position. Write that to the output file and read a new line from this file.	[reply]