Re: adaptive syslog message parsing

Maybe this doesn't exactly answer the question, but should maybe be asked...

Personally I love writing code, but I'm much more efficient if I can beg, borrow, or well I guess I won't steal but you get the drift...

First, since this is "from a variety of sources/daemons, in a variety of formats", I'd consider configuring the syslog to dump different sources of logs to different files. Then you limit the parsing issues to a particular topic... you should not have to deal with parsing mail log messages and dhcp messages in the same file.

Second you note the volume and "who wants to read every line". I would suggest you consider something like rrdtool. It has many parsers for various kinds of log files and it makes pretty graphs. Even if you choose not to use rrdtool, you can grab the parsers, many of which are perl, and look at them for parsing each of the various formats in which you are interested... (Cavat... I have not used rrdtool yet, but am planning on it.)

..Otto

Comment on Re: adaptive syslog message parsing

Replies are listed 'Best First'.
Re^2: adaptive syslog message parsing by Anonymous Monk on Jun 07, 2007 at 17:53 UTC
thanks for the ideas otto.. unfortunately i have neither the access nor the authority to start modifying the syslog server itself since it's in production.. i'll have the server's architect take your suggestion into consideration, but it may cause regressive problems with other systems that rely on the way it's structured now (albeit i'll agree it isn't very well designed)... as far as rrdtool, i've used it a bit.. it definitely makes nice graphs, but it has more of a numerical-analysis application rather than dealing with text parsing.. it's, at its core, a backend db for storing data in a fixed amount of spare, along with extensions to create graphs from the db you populated with (usually) numerical measurements, counters, and the like..	[reply]

Replies are listed 'Best First'.

Re^2: adaptive syslog message parsing
by Anonymous Monk on Jun 07, 2007 at 17:53 UTC

as far as rrdtool, i've used it a bit.. it definitely makes nice graphs, but it has more of a numerical-analysis application rather than dealing with text parsing.. it's, at its core, a backend db for storing data in a fixed amount of spare, along with extensions to create graphs from the db you populated with (usually) numerical measurements, counters, and the like..

[reply]