Parse log file

pr33 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I have a log file which needs o/p in the format below. For each of the time stamp, I need to print out the name of the time stamp, The no. of times it appeared in the logfile as well as the no. of times the message appeared for a service for the given time stamp. Here is my sample input file. The log file starts with time stamp followed by hostname and then the service name. While accessing the Data structure, I am seeing an incorrect count for the time stamp and 0 values for the service count corresponding to the time stamp. Please provide your feedback or suggestions here.

Here are the lines of input.

Jan  7 04:13:02 centos1 rsyslogd: [origin software="rsyslogd" swVersio
+n="5.8.10" x-pid="1453" x-info="http://www.rsyslog.com"] rsyslogd was
+ HUPed
Jan  7 04:15:41 centos1 puppet-agent[5467]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 04:15:41 centos1 puppet-agent[5467]: getaddrinfo: Name or servi
+ce not known
Jan  7 04:15:46 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 04:15:51 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 04:15:56 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 04:16:01 centos1 puppet-agent[5467]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 04:16:39 centos1 puppet-agent[5467]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 04:16:39 centos1 puppet-agent[5467]: Using cached catalog
Jan  7 04:16:39 centos1 puppet-agent[5467]: We have matched the Centos
+ Agent Node
Jan  7 04:16:39 centos1 puppet-agent[5467]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 04:16:39 centos1 puppet-agent[5467]: Finished catalog run in 0.
+19 seconds
Jan  7 04:16:44 centos1 puppet-agent[5467]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 05:15:41 centos1 puppet-agent[6419]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 05:15:41 centos1 puppet-agent[6419]: getaddrinfo: Name or servi
+ce not known
Jan  7 05:15:46 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 05:15:51 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 05:15:56 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 05:16:01 centos1 puppet-agent[6419]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 05:16:39 centos1 puppet-agent[6419]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 05:16:39 centos1 puppet-agent[6419]: Using cached catalog
Jan  7 05:16:39 centos1 puppet-agent[6419]: We have matched the Centos
+ Agent Node
Jan  7 05:16:39 centos1 puppet-agent[6419]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 05:16:39 centos1 puppet-agent[6419]: Finished catalog run in 0.
+18 seconds
Jan  7 05:16:44 centos1 puppet-agent[6419]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 06:15:41 centos1 puppet-agent[6831]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 06:15:41 centos1 puppet-agent[6831]: getaddrinfo: Name or servi
+ce not known
Jan  7 06:15:46 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 06:15:51 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 06:15:56 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 06:16:01 centos1 puppet-agent[6831]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 06:16:39 centos1 puppet-agent[6831]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 06:16:39 centos1 puppet-agent[6831]: Using cached catalog
Jan  7 06:16:39 centos1 puppet-agent[6831]: We have matched the Centos
+ Agent Node
Jan  7 06:16:39 centos1 puppet-agent[6831]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 06:16:39 centos1 puppet-agent[6831]: Finished catalog run in 0.
+16 seconds
Jan  7 06:16:44 centos1 puppet-agent[6831]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 07:15:41 centos1 puppet-agent[7141]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 07:15:41 centos1 puppet-agent[7141]: getaddrinfo: Name or servi
+ce not known
Jan  7 07:15:46 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 07:15:51 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 07:15:56 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 07:16:01 centos1 puppet-agent[7141]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 07:16:39 centos1 puppet-agent[7141]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 07:16:39 centos1 puppet-agent[7141]: Using cached catalog
Jan  7 07:16:39 centos1 puppet-agent[7141]: We have matched the Centos
+ Agent Node
Jan  7 07:16:39 centos1 puppet-agent[7141]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 07:16:39 centos1 puppet-agent[7141]: Finished catalog run in 0.
+16 seconds
Jan  7 07:16:44 centos1 puppet-agent[7141]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 08:15:41 centos1 puppet-agent[7450]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 08:15:41 centos1 puppet-agent[7450]: getaddrinfo: Name or servi
+ce not known
Jan  7 08:15:46 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 08:15:51 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 08:15:56 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 08:16:01 centos1 puppet-agent[7450]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 08:16:38 centos1 puppet-agent[7450]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 08:16:38 centos1 puppet-agent[7450]: Using cached catalog
Jan  7 08:16:39 centos1 puppet-agent[7450]: We have matched the Centos
+ Agent Node
Jan  7 08:16:39 centos1 puppet-agent[7450]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 08:16:39 centos1 puppet-agent[7450]: Finished catalog run in 0.
+16 seconds
Jan  7 08:16:44 centos1 puppet-agent[7450]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 09:15:41 centos1 puppet-agent[7760]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 09:15:41 centos1 puppet-agent[7760]: getaddrinfo: Name or servi
+ce not known
Jan  7 09:15:46 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 09:15:51 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 09:15:56 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 09:16:01 centos1 puppet-agent[7760]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 09:16:38 centos1 puppet-agent[7760]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 09:16:38 centos1 puppet-agent[7760]: Using cached catalog
Jan  7 09:16:39 centos1 puppet-agent[7760]: We have matched the Centos
+ Agent Node
Jan  7 09:16:39 centos1 puppet-agent[7760]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 09:16:39 centos1 puppet-agent[7760]: Finished catalog run in 0.
+16 seconds
Jan  7 09:16:44 centos1 puppet-agent[7760]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 10:15:41 centos1 puppet-agent[8075]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 10:15:41 centos1 puppet-agent[8075]: getaddrinfo: Name or servi
+ce not known
Jan  7 10:15:46 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 10:15:51 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 10:15:56 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 10:16:01 centos1 puppet-agent[8075]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 10:16:39 centos1 puppet-agent[8075]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 10:16:39 centos1 puppet-agent[8075]: Using cached catalog
Jan  7 10:16:39 centos1 puppet-agent[8075]: We have matched the Centos
+ Agent Node
Jan  7 10:16:39 centos1 puppet-agent[8075]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 10:16:39 centos1 puppet-agent[8075]: Finished catalog run in 0.
+16 seconds
Jan  7 10:16:44 centos1 puppet-agent[8075]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 11:15:41 centos1 puppet-agent[8391]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 11:15:41 centos1 puppet-agent[8391]: getaddrinfo: Name or servi
+ce not known
Jan  7 11:15:46 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 11:15:51 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 11:15:56 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 11:16:01 centos1 puppet-agent[8391]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 11:16:38 centos1 puppet-agent[8391]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 11:16:38 centos1 puppet-agent[8391]: Using cached catalog
Jan  7 11:16:39 centos1 puppet-agent[8391]: We have matched the Centos
+ Agent Node
Jan  7 11:16:39 centos1 puppet-agent[8391]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 11:16:39 centos1 puppet-agent[8391]: Finished catalog run in 0.
+16 seconds
Jan  7 11:16:44 centos1 puppet-agent[8391]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 12:15:41 centos1 puppet-agent[8847]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 12:15:41 centos1 puppet-agent[8847]: getaddrinfo: Name or servi
+ce not known
Jan  7 12:15:46 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 12:15:51 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 12:15:56 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 12:16:01 centos1 puppet-agent[8847]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 12:16:39 centos1 puppet-agent[8847]: Could not retrieve catalog
+ from remote server: getaddrinfo: Name or service not known
Jan  7 12:16:39 centos1 puppet-agent[8847]: Using cached catalog
Jan  7 12:16:39 centos1 puppet-agent[8847]: We have matched the Centos
+ Agent Node
Jan  7 12:16:39 centos1 puppet-agent[8847]: (/Stage[main]/Main/Node[ce
+ntos]/Notify[We have matched the Centos Agent Node]/message) defined 
+'message' as 'We have matched the Centos Agent Node'
Jan  7 12:16:39 centos1 puppet-agent[8847]: Finished catalog run in 0.
+18 seconds
Jan  7 12:16:44 centos1 puppet-agent[8847]: Could not send report: get
+addrinfo: Name or service not known
Jan  7 13:15:41 centos1 puppet-agent[9305]: Unable to fetch my node de
+finition, but the agent run will continue:
Jan  7 13:15:41 centos1 puppet-agent[9305]: getaddrinfo: Name or servi
+ce not known
Jan  7 13:15:43 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168
+.0.1 port 67 (xid=0xf27d6e5)
Jan  7 13:15:46 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168
+.0.1 port 67 (xid=0xf27d6e5)
Jan  7 13:15:46 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/fac
+ts.d]) Failed to generate additional resources using 'eval_generate':
+ getaddrinfo: Name or service not known
Jan  7 13:15:51 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/fac
+ts.d]) Could not evaluate: Could not retrieve file metadata for puppe
+t://puppet/pluginfacts: getaddrinfo: Name or service not known
Jan  7 13:15:52 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168
+.0.1 port 67 (xid=0xf27d6e5)
Jan  7 13:15:56 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/lib
+]) Failed to generate additional resources using 'eval_generate': get
+addrinfo: Name or service not known
Jan  7 13:16:01 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/lib
+]) Could not evaluate: Could not retrieve file metadata for puppet://
+puppet/plugins: getaddrinfo: Name or service not known
Jan  7 13:16:02 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168
+.0.1 port 67 (xid=0xf27d6e5)
Jan  7 13:16:15 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168
+.0.1 port 67 (xid=0xf27d6e5)
[download]

Expected Output(Sample for few time stamps).

Time Number_of_times_timestamp_appeared rsyslogd puppet-agent dhclient

Jan 7 04:16:39 5 0 5 0

Jan 7 13:15:43 1 0 0 1

Jan 7 04:13:02 1 1 0 0

Here is my code. I am storing the data as Hash of an Arrays by reading the Log file

#!/usr/bin/perl
use warnings;
use strict;
use Data::Dumper;
###########################

open my $fh, '<', 'messages.txt' or die;
my %data;
my %seen;
my @services;
while (<$fh>) {
    chomp;
    if (/(^\w+\s+\d+\s+\d+:\d+:\d+)\s+\w+\s+(.*?)(\[[0-9]+\]:|:)/) {
        push @{$data{$1}}, [$2];
        push @services, $2;
    }

}
close($fh);

#print Dumper \%data;

my @uniq = grep { ! $seen{$_}++ } @services;
print "Time_Stamp  Count   @uniq\n";
my $cnt = {};
foreach my $k (keys %data) {
        my $count = scalar(@{$data{$k}});
        foreach my $s (@uniq) {
        foreach my $i (0..$#{$data{$k}}) {
            if ($s eq ${$data{$k}}[$i]) {
            $cnt->{$s}++;
           } else {
            $cnt->{$s} = 0;
          }
     }
     }
    print "$k $count", join("        ", values %$cnt), "\n";
    $cnt = {};
 }
[download]

Partial Data Dumper O/p.

                               ],
          'Jan  7 07:16:39' => [
                                 [
                                   'puppet-agent'
                                 ],
                                 [
                                   'puppet-agent'
                                 ],
                                 [
                                   'puppet-agent'
                                 ],
                                 [
                                   'puppet-agent'
                                 ],
                                 [
                                   'puppet-agent'
                                 ]
                               ],
          'Jan  7 11:16:39' => [
                                 [
                                   'puppet-agent'
                                 ],
                                 [
                                   'puppet-agent'
                                 ],
                                 [
                                   'puppet-agent'
                                 ]
                               ],
          'Jan  7 13:16:01' => [
                                 [
                                   'puppet-agent'
                                 ]
                               ],
           ..........
          .............
         ..........
[download]

Comment on Parse log file Select or Download Code

Replies are listed 'Best First'.
Re: Parse log file by tybalt89 (Monsignor) on Jan 10, 2018 at 13:45 UTC
#!/usr/bin/perl # http://perlmonks.org/?node_id=1207042 use strict; use warnings; my (%services, %bydate); while(<DATA>) { /^(.{15}) (\S+) ([\w-]+)/ or next; $services{$3}++; $bydate{$1}{count}++; $bydate{$1}{$3}++; } my @services = reverse sort keys %services; print "Time Number_of_times_timestamp_appeared @services\n"; for my $key ( sort keys %bydate ) { print "$key $bydate{$key}{count} @{[ map $bydate{$key}{$_} // 0, @services ]}\n"; } __DATA__ Jan 7 04:13:02 centos1 rsyslogd: [origin software="rsyslogd" swVersio +n="5.8.10" x-pid="1453" x-info="http://www.rsyslog.com"] rsyslogd was + HUPed Jan 7 04:15:41 centos1 puppet-agent[5467]: Unable to fetch my node de +finition, but the agent run will continue: Jan 7 04:15:41 centos1 puppet-agent[5467]: getaddrinfo: Name or servi +ce not known Jan 7 13:16:01 centos1 puppet-agent[9305]: (/File[/var/lib/puppet/lib +]) Could not evaluate: Could not retrieve file metadata for puppet:// +puppet/plugins: getaddrinfo: Name or service not known Jan 7 13:16:02 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5) Jan 7 13:16:15 centos1 dhclient[1624]: DHCPREQUEST on eth0 to 192.168 +.0.1 port 67 (xid=0xf27d6e5) [download] Outputs: `Time Number_of_times_timestamp_appeared rsyslogd puppet-agent dhclient Jan 7 04:13:02 1 1 0 0 Jan 7 04:15:41 2 0 2 0 Jan 7 13:16:01 1 0 1 0 Jan 7 13:16:02 1 0 0 1 Jan 7 13:16:15 1 0 0 1` [download]	[reply] [d/l] [select]
Re^2: Parse log file by pr33 (Scribe) on Jan 11, 2018 at 02:51 UTC
Thanks . It works . Doing this by generating a Multi-Dimensional Hash seems to be more fairer than Hash of Arrays.	[reply]