Hello Monks, I have a log file which needs o/p in the format below. For each of the time stamp, I need to print out the name of the time stamp, The no. of times it appeared in the logfile as well as the no. of times the message appeared for a service for the given time stamp. Here is my sample input file. The log file starts with time stamp followed by hostname and then the service name. While accessing the Data structure, I am seeing an incorrect count for the time stamp and 0 values for the service count corresponding to the time stamp. Please provide your feedback or suggestions here.
Here are the lines of input.
Jan 7 04:13:02 centos1 rsyslogd: [origin software="rsyslogd" swVersio +n="5.8.10" x-pid="1453" x-info="http://www.rsyslog.com"] rsyslogd was + HUPed Jan 7 04:15:41 centos1 puppet-agent[5467]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 04:15:41 centos1 puppet-agent[5467]: getaddrinfo: Name or servi +ce not known Jan 7 04:15:46 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 04:15:51 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 04:15:56 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 04:16:01 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 04:16:39 centos1 puppet-agent[5467]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 04:16:39 centos1 puppet-agent[5467]: Using cached catalog Jan 7 04:16:39 centos1 puppet-agent[5467]: We have matched the Centos + Agent Node Jan 7 04:16:39 centos1 puppet-agent[5467]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 04:16:39 centos1 puppet-agent[5467]: Finished catalog run in 0. +19 seconds Jan 7 04:16:44 centos1 puppet-agent[5467]: Could not send report: get +addrinfo: Name or service not known Jan 7 05:15:41 centos1 puppet-agent[6419]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 05:15:41 centos1 puppet-agent[6419]: getaddrinfo: Name or servi +ce not known Jan 7 05:15:46 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 05:15:51 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 05:15:56 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 05:16:01 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 05:16:39 centos1 puppet-agent[6419]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 05:16:39 centos1 puppet-agent[6419]: Using cached catalog Jan 7 05:16:39 centos1 puppet-agent[6419]: We have matched the Centos + Agent Node Jan 7 05:16:39 centos1 puppet-agent[6419]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 05:16:39 centos1 puppet-agent[6419]: Finished catalog run in 0. +18 seconds Jan 7 05:16:44 centos1 puppet-agent[6419]: Could not send report: get +addrinfo: Name or service not known Jan 7 06:15:41 centos1 puppet-agent[6831]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 06:15:41 centos1 puppet-agent[6831]: getaddrinfo: Name or servi +ce not known Jan 7 06:15:46 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 06:15:51 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 06:15:56 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 06:16:01 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 06:16:39 centos1 puppet-agent[6831]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 06:16:39 centos1 puppet-agent[6831]: Using cached catalog Jan 7 06:16:39 centos1 puppet-agent[6831]: We have matched the Centos + Agent Node Jan 7 06:16:39 centos1 puppet-agent[6831]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 06:16:39 centos1 puppet-agent[6831]: Finished catalog run in 0. +16 seconds Jan 7 06:16:44 centos1 puppet-agent[6831]: Could not send report: get +addrinfo: Name or service not known Jan 7 07:15:41 centos1 puppet-agent[7141]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 07:15:41 centos1 puppet-agent[7141]: getaddrinfo: Name or servi +ce not known Jan 7 07:15:46 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 07:15:51 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 07:15:56 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 07:16:01 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 07:16:39 centos1 puppet-agent[7141]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 07:16:39 centos1 puppet-agent[7141]: Using cached catalog Jan 7 07:16:39 centos1 puppet-agent[7141]: We have matched the Centos + Agent Node Jan 7 07:16:39 centos1 puppet-agent[7141]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 07:16:39 centos1 puppet-agent[7141]: Finished catalog run in 0. +16 seconds Jan 7 07:16:44 centos1 puppet-agent[7141]: Could not send report: get +addrinfo: Name or service not known Jan 7 08:15:41 centos1 puppet-agent[7450]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 08:15:41 centos1 puppet-agent[7450]: getaddrinfo: Name or servi +ce not known Jan 7 08:15:46 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 08:15:51 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 08:15:56 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 08:16:01 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 08:16:38 centos1 puppet-agent[7450]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 08:16:38 centos1 puppet-agent[7450]: Using cached catalog Jan 7 08:16:39 centos1 puppet-agent[7450]: We have matched the Centos + Agent Node Jan 7 08:16:39 centos1 puppet-agent[7450]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 08:16:39 centos1 puppet-agent[7450]: Finished catalog run in 0. +16 seconds Jan 7 08:16:44 centos1 puppet-agent[7450]: Could not send report: get +addrinfo: Name or service not known Jan 7 09:15:41 centos1 puppet-agent[7760]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 09:15:41 centos1 puppet-agent[7760]: getaddrinfo: Name or servi +ce not known Jan 7 09:15:46 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 09:15:51 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 09:15:56 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 09:16:01 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 09:16:38 centos1 puppet-agent[7760]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 09:16:38 centos1 puppet-agent[7760]: Using cached catalog Jan 7 09:16:39 centos1 puppet-agent[7760]: We have matched the Centos + Agent Node Jan 7 09:16:39 centos1 puppet-agent[7760]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 09:16:39 centos1 puppet-agent[7760]: Finished catalog run in 0. +16 seconds Jan 7 09:16:44 centos1 puppet-agent[7760]: Could not send report: get +addrinfo: Name or service not known Jan 7 10:15:41 centos1 puppet-agent[8075]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 10:15:41 centos1 puppet-agent[8075]: getaddrinfo: Name or servi +ce not known Jan 7 10:15:46 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 10:15:51 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 10:15:56 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 10:16:01 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 10:16:39 centos1 puppet-agent[8075]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 10:16:39 centos1 puppet-agent[8075]: Using cached catalog Jan 7 10:16:39 centos1 puppet-agent[8075]: We have matched the Centos + Agent Node Jan 7 10:16:39 centos1 puppet-agent[8075]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 10:16:39 centos1 puppet-agent[8075]: Finished catalog run in 0. +16 seconds Jan 7 10:16:44 centos1 puppet-agent[8075]: Could not send report: get +addrinfo: Name or service not known Jan 7 11:15:41 centos1 puppet-agent[8391]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 11:15:41 centos1 puppet-agent[8391]: getaddrinfo: Name or servi +ce not known Jan 7 11:15:46 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 11:15:51 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 11:15:56 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 11:16:01 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 11:16:38 centos1 puppet-agent[8391]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 11:16:38 centos1 puppet-agent[8391]: Using cached catalog Jan 7 11:16:39 centos1 puppet-agent[8391]: We have matched the Centos + Agent Node Jan 7 11:16:39 centos1 puppet-agent[8391]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 11:16:39 centos1 puppet-agent[8391]: Finished catalog run in 0. +16 seconds Jan 7 11:16:44 centos1 puppet-agent[8391]: Could not send report: get +addrinfo: Name or service not known Jan 7 12:15:41 centos1 puppet-agent[8847]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 12:15:41 centos1 puppet-agent[8847]: getaddrinfo: Name or servi +ce not known Jan 7 12:15:46 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 12:15:51 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 12:15:56 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 12:16:01 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 12:16:39 centos1 puppet-agent[8847]: Could not retrieve catalog + from remote server: getaddrinfo: Name or service not known Jan 7 12:16:39 centos1 puppet-agent[8847]: Using cached catalog Jan 7 12:16:39 centos1 puppet-agent[8847]: We have matched the Centos + Agent Node Jan 7 12:16:39 centos1 puppet-agent[8847]: (/Stage[main]/Main/Node[ce +ntos]/Notify[We have matched the Centos Agent Node]/message) defined +'message' as 'We have matched the Centos Agent Node' Jan 7 12:16:39 centos1 puppet-agent[8847]: Finished catalog run in 0. +18 seconds Jan 7 12:16:44 centos1 puppet-agent[8847]: Could not send report: get +addrinfo: Name or service not known Jan 7 13:15:41 centos1 puppet-agent[9305]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 13:15:41 centos1 puppet-agent[9305]: getaddrinfo: Name or servi +ce not known Jan 7 13:15:43 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5) Jan 7 13:15:46 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5) Jan 7 13:15:46 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/fac +ts.d]) Failed to generate additional resources using 'eval_generate': + getaddrinfo: Name or service not known Jan 7 13:15:51 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/fac +ts.d]) Could not evaluate: Could not retrieve file metadata for puppe +t://puppet/pluginfacts: getaddrinfo: Name or service not known Jan 7 13:15:52 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5) Jan 7 13:15:56 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/lib +]) Failed to generate additional resources using 'eval_generate': get +addrinfo: Name or service not known Jan 7 13:16:01 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 13:16:02 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5) Jan 7 13:16:15 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5)
Expected Output(Sample for few time stamps).
Time Number_of_times_timestamp_appeared rsyslogd puppet-agent dhclient
Jan 7 04:16:39 5 0 5 0
Jan 7 13:15:43 1 0 0 1
Jan 7 04:13:02 1 1 0 0
Here is my code. I am storing the data as Hash of an Arrays by reading the Log file
#!/usr/bin/perl use warnings; use strict; use Data::Dumper; ########################### open my $fh, '<', 'messages.txt' or die; my %data; my %seen; my @services; while (<$fh>) { chomp; if (/(^\w+\s+\d+\s+\d+:\d+:\d+)\s+\w+\s+(.*?)(\[[0-9]+\]:|:)/) { push @{$data{$1}}, [$2]; push @services, $2; } } close($fh); #print Dumper \%data; my @uniq = grep { ! $seen{$_}++ } @services; print "Time_Stamp Count @uniq\n"; my $cnt = {}; foreach my $k (keys %data) { my $count = scalar(@{$data{$k}}); foreach my $s (@uniq) { foreach my $i (0..$#{$data{$k}}) { if ($s eq ${$data{$k}}[$i]) { $cnt->{$s}++; } else { $cnt->{$s} = 0; } } } print "$k $count", join(" ", values %$cnt), "\n"; $cnt = {}; }
Partial Data Dumper O/p.
], 'Jan 7 07:16:39' => [ [ 'puppet-agent' ], [ 'puppet-agent' ], [ 'puppet-agent' ], [ 'puppet-agent' ], [ 'puppet-agent' ] ], 'Jan 7 11:16:39' => [ [ 'puppet-agent' ], [ 'puppet-agent' ], [ 'puppet-agent' ] ], 'Jan 7 13:16:01' => [ [ 'puppet-agent' ] ], .......... ............. ..........
In reply to Parse log file by pr33
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |