Re: adaptive syslog message parsing

Given that you could have many types of logged messages, with each having its own format specification, you might apply a regex to determine its type. Once an entry is classified, send it to an appropriate handler (subroutine) that knows how to parse that type of entry into its component parts, and then stuff the guts you care about into an appropriate data structure.

The structure might look like:

a hash of servers
  a hash of daemon names
    a hash of messages (and their cumulative frequencies)
[download]

For entries that do not classify to a handler you have coded, send these to an exception report (log file), and create the necessary handlers later as needed.

Where do you want *them* to go today?

Comment on Re: adaptive syslog message parsing Download Code

Replies are listed 'Best First'.
Re^2: adaptive syslog message parsing by neosamuri (Friar) on Jun 07, 2007 at 04:47 UTC
It seems to me the cumulative frequencies of message types(for each server, daemon name combination), would be less useful then the sequence of events. `hash{servers} hash{daemon names} array[$time, message] or $struct->{$server}->{$daemon}->[$message_id] = [$time,$message];` [download] Though for either method it would be nice to send a GD generated graph for each set server/daemon, which can give quick access to needed information.	[reply] [d/l]
Re^3: adaptive syslog message parsing by thezip (Vicar) on Jun 07, 2007 at 07:12 UTC
Yes, these are a nice enhancements, but they are not to the OP's specification which detailed that the statistical frequencies (ie. counts) be stored in the lowest hashes. I cannot comment as to which is better since I don't have the bigger picture of the problem. Where do you want them* to go today?*	[reply]