Dealing with Syslog files

stevbutt has asked for the wisdom of the Perl Monks concerning the following question:

Thanks for the suggestions, looks to me like Syslog is just too variable however I have another idea

My lines look like the following

May 2 04:06:15 lon-pop.mail.mydom.com pop3login: LOGOUT, user=gonenow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=5560, time=1

The first three parts are pretty standard i.e. datetime( though with no year ) sysloghost, operation then the variable message.

What I would like to do is convert this to a csv line as follows where its broken into at least four parts but if the string user=gonenow appears then add gonenow ( or whatever the users name was ) as an extra field, same for ip=

02/05/2012 04:06:15,lon-pop.mail.mydom.com,pop3login,"LOGOUT, user=gonenow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=5560, time=1</c>",gonenow,::ffff:127.0.0.1

Comment on Dealing with Syslog files Select or Download Code

Replies are listed 'Best First'.
Re: Dealing with Syslog files by zentara (Cardinal) on Jul 31, 2012 at 16:19 UTC
Hi, Syslogs are simple text files, which you can process line by line. You can regex or split the lines to extract your data. See parsing a syslog entry for example code. I'm not really a human, but I play one on earth. Old Perl Programmer Haiku ................... flash japh	[reply]
Re: Dealing with Syslog files by grumbert (Scribe) on Jul 31, 2012 at 16:22 UTC
Well I can confirm it works very well for Solaris 8/9, we use it in a medium size config (~ 3 million log messages / day) in conjunction with File::Tail and it tends to be very stable. The question is really more how variable the input is. The syslog format is hardly standard and I've seen it altered significantly with syslog-ng, to the point where it was just easier to flex your regex muscles ;)	[reply]
Re: Dealing with Syslog files by cdukes (Initiate) on Aug 02, 2012 at 13:53 UTC
Here's a working example for converting logs to LogZilla format (which pipes messages into a DB using tab separated values). You could easily adapt it for your needs I'm sure. http://forum.logzilla.pro/index.php/topic,439.0.html	[reply]