Maybe this doesn't exactly answer the question, but should maybe be asked...

Personally I love writing code, but I'm much more efficient if I can beg, borrow, or well I guess I won't steal but you get the drift...

First, since this is "from a variety of sources/daemons, in a variety of formats", I'd consider configuring the syslog to dump different sources of logs to different files. Then you limit the parsing issues to a particular topic... you should not have to deal with parsing mail log messages and dhcp messages in the same file.

Second you note the volume and "who wants to read every line". I would suggest you consider something like rrdtool. It has many parsers for various kinds of log files and it makes pretty graphs. Even if you choose not to use rrdtool, you can grab the parsers, many of which are perl, and look at them for parsing each of the various formats in which you are interested... (Cavat... I have not used rrdtool yet, but am planning on it.)

..Otto

In reply to Re: adaptive syslog message parsing by otto
in thread adaptive syslog message parsing by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.