Re: Syslog files revisited

split and sprintf will probably do what you want.

knoppix@Microknoppix:~$ perl -Mstrict -Mwarnings -E '
> my $line = q{May  2 04:06:15 lon-pop.mail.mydom.com pop3login: LOGOU
+T, user=gonenow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=
+5560, time=1};
> my ( $mon, $day, $time, $dom, $login, $remainder ) =
>    split m{:?\s+}, $line, 6;
> my %monthNos = do {
>    my $no = 0;
>    map { $_ => ++ $no }
>    qw{ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec };
>    };
> 
> my $yr = q{2012};
> my $csv = sprintf q{%02d/%02d/%s %s,%s,%s,%s},
>    $day, $monthNos{ $mon }, $yr, $time, $dom, $login, $remainder;
> 
> say $csv;'
02/05/2012 04:06:15,lon-pop.mail.mydom.com,pop3login,LOGOUT, user=gone
+now, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=5560, time=1
knoppix@Microknoppix:~$
[download]

I hope this is helpful.

Update: After replying to your reply I realised I had totally missed the need to quote the variable message and add the user and ip to the csv line. Here is revised code.

knoppix@Microknoppix:~$ perl -Mstrict -Mwarnings -E '
> my $line = q{May  2 04:06:15 lon-pop.mail.mydom.com pop3login: LOGOU
+T,  user=gonenow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent
+=5560, time=1};
> my ( $mon, $day, $time, $dom, $login, $remainder ) =
>    split m{:?\s+}, $line, 6;
> my %monthNos = do {
>    my $no = 0;
>    map { $_ => ++ $no }
>    qw{ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec };
>    };
> 
> my $yr = q{2012};
> my ( $user, $ip ) =
>    $remainder =~ m{user=([^,]+),\s+ip=\[([^\]]+)};
> $remainder = qq{"$remainder"};
> my $csv = sprintf q{%02d/%02d/%s %s,%s,%s,%s,%s,%s},
>    $day, $monthNos{ $mon }, $yr, $time, $dom,
>    $login, $remainder, $user, $ip;
> 
> say $csv;'
02/05/2012 04:06:15,lon-pop.mail.mydom.com,pop3login,"LOGOUT, user=gon
+enow, ip=[::ffff:127.0.0.1], top=0, retr=0, rcvd=24, sent=5560, time=
+1",gonenow,::ffff:127.0.0.1
knoppix@Microknoppix:~$
[download]

Cheers,

JohnGG

Comment on Re: Syslog files revisited Select or Download Code

Replies are listed 'Best First'.
Re^2: Syslog files revisited by stevbutt (Novice) on Aug 02, 2012 at 23:15 UTC
This is certainly a move in the right direction and the date part is superb !, but it gives me the problem that the variable message at the end gets split up into an indeterminate number of csv values where if it were at least quoted I could read that as one field as the commas inside the quotes would be ignored ( I think )	[reply]
Re^3: Syslog files revisited by johngg (Canon) on Aug 03, 2012 at 09:11 UTC
... but it gives me the problem that the variable message at the end gets split up into an indeterminate number of csv values ... No, the third parameter to split limits the number of resultant fields so the `$remainder` scalar variable holds the entirety of your variable message. Cheers, JohnGG	[reply] [d/l]
Re^4: Syslog files revisited by stevbutt (Novice) on Aug 03, 2012 at 16:09 UTC
Thats great and thank you very much for your help. I have one further issue though. My existing code now looks like this `#!/usr/bin/perl use 5.010; use strict; use warnings; while (my $line = <STDIN>) { chomp($line); my ( $mon, $day, $time, $dom, $login, $remainder ) = split m{:?\s+}, $line, 6; my %monthNos = do { my $no = 0; map { $_ => ++ $no } qw{ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec }; }; my $yr = q{2012}; my $csv = sprintf q{%02d/%02d/%s %s,%s,%s,"%s"}, $day, $monthNos{ $mon }, $yr, $time, $dom, $login, $remainder; say $csv; }` [download] As I wanted the remainder to be in double quotes so it would read in as one field. The problem is that remainder sometimes already contains double quotes which I would like to remove or replace with single quotes I tried the following but it leaves be with an empty set of double quotes `my $remainder =~ s/""\"\""/\'/g;`	[reply] [d/l] [select]
Re^5: Syslog files revisited by stevbutt (Novice) on Aug 03, 2012 at 16:18 UTC
Re^2: Syslog files revisited by stevbutt (Novice) on Aug 10, 2012 at 13:51 UTC
Thanks John, Almost all lines are now being loaded correctly but I have one further problem with occasional lines and specifically just on the user= part. Sometimes it says this `user=gonenow sent=34 recv=34` Or `user=<gonenow>, etc etc` Instead of the comma separated list I expected Is there a way to just extract my user value up until either a comma or space and to strip < > if they occur ?	[reply] [d/l] [select]
Re^3: Syslog files revisited by johngg (Canon) on Aug 10, 2012 at 19:06 UTC
The `user=<gonenow>, etc etc` case can probably be got around with a couple of minor changes to the code. The `user=gonenow sent=34 recv=34` is more of a problem as it fails the premise on which the split approach was based. I'd probably wrap the original code in an `if` condition and deal with the non-comma-separated lines in an `else` clause. It just so happens that I am about to go on holiday so I will not be able to provide further help for a week or so. I suggest you create another question in SoPW, linking to this thread, and post some example data lines illustrating your problem so that others can perhaps help you build on the solution you have so far. Cheers, JohnGG	[reply] [d/l] [select]