in reply to Re: adaptive syslog message parsing
in thread adaptive syslog message parsing

Ok, I couldn't resist!

add:

use Algorithm::Diff;

toward the start. In sub add change:

push @{$context->{tails}}, $line;

to:

push @{$context->{tails}}, [$line =~ /(\S+)/g];

In sub mergeTails replace everything after:

my @groups;

with:

push @{$groups[@$_]}, $_ for @tails; @groups = grep {defined $_} @groups; for my $group (@groups) { my @ref = @{$group->[-1]}; my @org = @ref; my $count = 1; pop @$group; while (@$group) { my @new = @{pop @$group}; my @diffs = Algorithm::Diff::diff (\@ref, \@new); for my $change (@diffs) { next unless $change->[0][0] eq '-'; $ref[$change->[0][1]] = undef; } ++$count; } for (0 .. $#ref) { next if defined $ref[$_]; $org[$_] = '*****'; } push @{$context->{digest}}, [join (' ', @org), $count]; }

Now prints:

infocache02 ldap_cachemgr (1) Error: Unable to refresh from profile:tls_automount_profil +e. (error=1) (1) libsldap: Status: 91 Mesg: openConnection: simple bind fai +led - Can't connect to the LDAP server sendmail (3) ***** Losing ***** savemail panic (2) ***** SYSERR(root): savemail: cannot save rejected email a +nywhere mail2-in postfix/smtpd (2) warning: 84.9.96.201: address not listed for hostname mail +.intechcentre.com (4) warning: ***** hostname ***** verification failed: hostnam +e nor servname provided, or not known mail2-out ntpd (5) ***** Bad file descriptor postfix/smtp (1) warning: valid_hostname: empty hostname (1) warning: malformed domain name in resource data of MX reco +rd for hotmil.com: postfix/smtp[32282] (1) warning: numeric domain name in resource data of MX record + for uyahoo.com: 10.0.0.2

DWIM is Perl's answer to Gödel

Replies are listed 'Best First'.
Re^3: adaptive syslog message parsing
by Anonymous Monk on Jun 07, 2007 at 17:20 UTC
    i admit, i lol'd when i read 'i couldn't resist..' i couldn't duplicate the output with the sample data using algorithm diff, it was similar but the new lines were off.. additionally, i have a more complete set of data that it doesn't output anything but one line (with the number 6 in parenthesis).. it looks pretty promising on the short set of sample data but i think it's confused with the big set of data (which happens to use fqdn instead of just hostname)

      I've performed a little data cleansing before adding lines - omitting empty lines seems to be the main fix! I also changed from using undef to '' in the diff code (Algorithm::Diff seemed unhappy with undefs) and tidied up the output a little.

      Given the large data set prints in part:

      ... mail1-out.nyc.domain.com ntpd (169) ***** Bad file descriptor postfix/smtp (2) warning: malformed domain name in resource data of MX +record for ***** (32) warning: no MX host for ***** has a valid address rec +ord (18) warning: numeric domain name in resource data of MX r +ecord for ***** 127.0.1.50 (2) warning: valid_hostname: empty hostname postfix/smtpd (7) warning: Illegal address syntax from ***** in RCPT com +mand: <jane@lulu.co $> sm-mta (2) ***** SYSERR(root): ***** config error: mail loops bac +k to me (MX problem?) syslog-ng (1) Changing permissions on special file /dev/console ... mail2-out.nyc.domain.com ntpd (168) ***** Bad file descriptor postfix/smtp (2) warning: malformed domain name in resource data of MX +record for ***** (25) warning: numeric domain name in resource data of MX r +ecord for ***** 10.0.0.2 (2) warning: valid_hostname: empty hostname sm-mta (1) l55DmFcQ022740: SYSERR(root): localhost.fabulous.com. +config error: mail loops back to me (MX problem?) syslog-ng (1) Changing permissions on special file /dev/console mail2-out.sfc.domain.com postfix/smtp (61) warning: malformed domain name in resource data of MX + record for ***** (1) warning: no MX host for epm.net has a valid address re +cord (61) warning: valid_hostname: empty hostname

      DWIM is Perl's answer to Gödel