Logfile Summarize messages by time and export to CSV

kris1511 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks, I seek wisdom on following question


#Write a script which parses /var/log/messages and generates a CSV wit
+h two columns: minute, number_of_messages in sorted time order.
#---------- begin sample output ----------
#minute, number_of_messages
#Jan 20 03:25,2
#Jan 20 03:26,2
#Jan 20 03:30,2
#Jan 20 05:03,1
#Jan 20 05:20,1
#Jan 20 05:22,6
#---------- end sample output ------------"""
[download]

Associated Text file is as follows

Jan 20 03:25:08 fakehost logrotate: ALERT exited abnormally with [1]
Jan 20 03:25:09 fakehost run-parts(/etc/cron.daily)[20447]: finished l
+ogrotate
Jan 20 03:26:21 fakehost anacron[28969]: Job 'cron.daily' terminated
Jan 20 03:26:22 fakehost anacron[28969]: Normal exit (1 job run)
Jan 20 03:30:01 fakehost CROND[31462]: (root) CMD (/usr/lib64/sa/sa1 1
+ 1)
Jan 20 03:30:01 fakehost CROND[31461]: (root) CMD (/var/system/bin/sys
+-cmd -F > /dev/null 2>&1)
Jan 20 05:03:03 fakehost ntpd[3705]: synchronized to time.faux.biz, st
+ratum 2
Jan 20 05:20:01 fakehost rsyslogd: [origin software="rsyslogd" swVersi
+on="5.8.10" x-pid="20438" x-info="http://www.rsyslog.com"] start
Jan 20 05:22:04 fakehost cs3[31163]:  Q: ".../bin/rsync -LD ": symlink
+ has no referent: "/var/syscmds/fakehost/runit_scripts/etc/runit/serv
+ice/superImportantService/run"#012Q: ".../bin/rsync -LD ": rsync erro
+r: some files/attrs were not transferred (see previous errors) (code 
+23) at main.c(1039) [sender=3.0.6]
Jan 20 05:22:04 fakehost cs3[31163]:  I: Last 2 quoted lines were gene
+rated by "/usr/local/bin/rsync -LD --recursive --delete --password-fi
+le=/var/syscmds/modules/rsync_password /var/syscmds/fakehost syscmds@
+fakehost::syscmds_rsync"
Jan 20 05:22:08 fakehost cs3[31163]:  Q: ".../sbin/sv restart": ok: ru
+n: /export/service/cool-service: (pid 32323) 0s
Jan 20 05:22:08 fakehost cs3[31163]:  I: Last 1 quoted lines were gene
+rated by "/sbin/sv restart /export/service/cool-service"
Jan 20 05:22:09 fakehost cs3[31163]:  R: cs3:  The cool service on fak
+ehost does not appear to be communicating with the cool service leade
+r.  Automating a restart of the cool service in attempt to resolve th
+e communication problem.
Jan 20 05:22:37 fakehost ACCT_ADD: WARNING: Manifest /var/syscmds/inpu
+ts/config-general/doit.txt has been processed already, bailing
[download]

open(IN, "fixed.txt") or die "cannot open file $!";
my %hash ={};
my @arr;

while(my $row =<IN>){
    
    push @arr, unpack("x0 A12",$row );
    #print unpack("x0 A12",$row ),"\n";
}
$hash{$_}++ for @arr;

# Adding Header
my @heading = ("minute", "number of messages");

open (OUT, '>:encoding(UTF-8)', "outputfile.csv") or die $!;
$csv->print ( OUT, $_ ) for @heading;
$csv->print ( OUT, $_ ) for sort keys %hash;   # Error's Out
close OUT;
[download]

I am getting error Can't call method print on undefine value on following line : $csv->print ( OUT, $_ ) for sort keys %hash; # Error's Out Any idea. Thanks in Advance!!!

Comment on Logfile Summarize messages by time and export to CSV Select or Download Code

Replies are listed 'Best First'.
Re: Logfile Summarize messages by time and export to CSV by huck (Prior) on Aug 31, 2017 at 22:56 UTC
Starting with `use strict; use warnings;` [download] would probably give you a clue. Also adding these two statements `use Text::CSV; my $csv = Text::CSV->new ( { binary => 1 } ) # should set binary att +ribute. or die "Cannot use CSV: ".Text::CSV->error_diag ();` [download] Would help a lot too. see http://search.cpan.org/~ishigaki/Text-CSV-1.95/lib/Text/CSV.pm and have you considered what happens if the log has Jan 31 and Feb 1 dates in it? Gonna mess up your sort isnt it? Ive been known to use a variant of the technique used here http://cpansearch.perl.org/src/DSCHWEI/Parse-Syslog-1.10/lib/Parse/Syslog.pm to convert those kinda dates into epoch datetimes. Once its in that mode it is easy to manipulate the final output sort and style.	[reply] [d/l] [select]
Re: Logfile Summarize messages by time and export to CSV by Tux (Canon) on Sep 01, 2017 at 06:20 UTC
The error indicated that you did not instantiate the `$csv` object. As huck already said, that could be something like: `use Text::CSV_XS; my $csv = Text::CSV_XS->new ({ binary => 1, eol => "\r\n", auto_diag = +> 1 });` [download] (do not forget the `eol` attribute, because the default is undefined.) But with that your code still won't run, as you didn't get the arguments to the `print` method correct. It expects a file handle and an anonymous list. Assuming your `@heading` is a list of labels `$csv->print (OUT, \@heading);` [download] Then inside the loop, you flatten the content. I am not sure if you want that. As the code shows now, it won't run anyway. If the result is indeed just one entry per row, as your loop is now (wich is strange for a header with more than one label) `$csv->print (OUT, [ $_ ]) for @arr;` [download] but if your loop is to generate more fields per line `while (my $row = <IN>) { push @arr, [ unpack "x0 A12 ..." => $row ]; } # @arr now holds a list of array refs $csv->print (OUT, $_) for @arr;` [download] Enjoy, Have FUN! H.Merijn	[reply] [d/l] [select]