Problem Parsing Log Files

ajay.awachar has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

I'm working on a script to parse log file in which I am searching for Critical and Major alarms in logs which appears in last one hour from current system time. I want to display the unique alarms (with latest timestamp) along their count of occurence. I need to do it with two dimensional arrays in perl.

I have written code like this.

#!/bin/perl 

use strict;

my $major=0;
my $minor=0;
my $critical=0;
my $systime=0;
my $syshr = 0;
my $sysmin=0;
my $syssec=0;



$systime=`date '+%H:%M:%S'`;

($syshr,$sysmin,$syssec) = split(/:/,$systime);

printf("Hr: $syshr Min: $sysmin Sec: $syssec");


my $errorcount = 0;
my $k = 0;
my @logdesc;

chdir('/opt/vk/app/osa/ecp/current/log/') or die "chdir Failed  $! \n"
+;


### This is the array in which we are grepping Critical and Major alar
+ms

@logdesc=`egrep 'MAJOR|CRITICAL' AppOsaEcp.log | cut -f 1-2  -d "]" | 
+sort | uniq`;

# printf("##### 1st line $logdesc[0]\n");

my $arraycount = 0 ;

$arraycount=@logdesc;


my  @fields;
my @errorstring;



@fields=split(/\s+/,$logdesc[0]);

printf("\n\nCritical and Major errors observerd during last one hour a
+re : \n\n");

my $i=0;
my $loghr=0;
my $logmin=0;
my $logsec=0;
my @errorarray;
my $cnt=0;
my $s=0;
my $matchcount=0;

###### Parsing the log file to find alarms depending on time-stamp to 
+find erros in last one hour

for($i=0;$i<=$arraycount;$i++)
{



     (@fields) = split(/\s+/,$logdesc[$i]);
     ($loghr,$logmin,$logsec) = split(/:/,$fields[1]);





        if(($syshr == 0) && ($loghr == 23))
        {


                if( $logmin > $sysmin )
                {


                      if( $fields[3] =~ /MAJOR/ )
                      {
                                $major++;
                      }

                      if( $fields[3] =~ /CRITICAL/ )
                      {
                                $critical++;
                      }
                      if( $fields[3] =~ /MINOR/ )
                      {
                                $minor++;
                      }


                      $errorarray[$cnt++] = $fields[4];

                      for($s=0;$s<$cnt;$s++)
                      {
                                if( $errorarray[$s] eq "$fields[4]" )
                                {
                                        $matchcount++;
                                }
                      }

                      (@errorstring) = split(/]/,$logdesc[$i]);

                      if( $matchcount == 1 )
                      {
                            

                                   printf("$fields[0]  $fields[1]  $er
+rorstring[1]");

                      }

                }
        }


        if(( $syshr >= 0) && ($loghr >= 0))
        {
                my $dhr = 0;

                $dhr=$syshr-$loghr;



                if( $dhr == 1)
                {

            if( $logmin >= $sysmin )
                        {

                                 if( $fields[3] =~ /MAJOR/ )
                                 {
                                        $major++;
                                 }

                                 if( $fields[3] =~ /CRITICAL/ )
                                 {
                                        $critical++;
                                 }
                                 if( $fields[3] =~ /MINOR/ )
                                 {
                                        $minor++;
                                 }

                                 $errorarray[$cnt++] = $fields[4];

                                for($s=0;$s<$cnt;$s++)
                                {
                                        if( $errorarray[$s] eq "$field
+s[4]" )
                                        {
                                                $matchcount++;
                                        }
                                }

                                (@errorstring) = split(/]/,$logdesc[$i
+]);

                                if( $matchcount == 1 )
                                {
                                      
                                         printf("$fields[0]  $fields[1
+]  $errorstring[1]");

                                }
                        }

                 }

                if( $dhr == 0)
                {
                        if( $logmin < $sysmin )
                        {


                                if( $fields[3] =~ /MAJOR/ )
                                {
                                        $major++;
                                }

                                if( $fields[3] =~ /CRITICAL/ )
                                {
                                        $critical++;
                                }
                                if( $fields[3] =~  /MINOR/ )
                                {
                                        $minor++;
                                }

                                $errorarray[$cnt++] = $fields[6];

                                for($s=0;$s<$cnt;$s++)
                                {
                                        if( $errorarray[$s] eq "$field
+s[4]" )
                                        {
                                                $matchcount++;
                                        }
                                }

                                (@errorstring) = split(/]/,$logdesc[$i
+]);

                                if( $matchcount == 1 )
                                {
                                     
                                          printf("$fields[0]  $fields[
+1]  $errorstring[1]");


                                }


                        }
                }
        }
}

print "\n\nCount of Alarms in Last One Hour .............. \n\n";
print "CRITICAL = $critical\nMAJOR = $major\nMINOR = $minor \n\n";
[download]

Current Output

Critical and Major errors observerd during last one hour are :

2009-06-18  09:36:18,666   MAJOR com.vk.service.rating.server.task.Rat
+eRequestTask - RATING ERROR - chargingSessionId: 25999, chargingParam
+eters: Undefined: 9999,Item: ECP Postpaid_Prepaid,SubType: RateQuery,
+Spid: 53,Calling Party Number: 6768874971,Called Party Number: a126a8
+864884,Bearer Service: 00,BillNum: 6768874971,RatePlan: RP5301`53,Scp
+Id: 9999,UsageHist: ,DateTime: 20090618093618,FlatFee: 

Count of Alarms in Last One Hour ..............
CRITICAL = 0
MAJOR = 81
MINOR = 0
[download]

DESIRED Output

Critical and Major errors observerd during last one hour are :

Critical Alarms
2009-06-18  09:36:18,666   CRITICAL com.vk.service.rating.server.task.
+OrginatingCallContext ( * With lastest timestamp)
Count=4 

2009-06-18  09:36:18,666   CRITICAL com.vk.service.rating.server - Cal
+l Threshold reached
Cont=10

Major Alarms 

2009-06-18  09:36:18,666   MAJOR com.vk.service.rating.server.task.Rat
+eRequestTask - RATING ERROR - chargingSessionId: 25999, chargingParam
+eters:: 0,FCA: true,**********SPLIT RATING PARAMETER**********Service
+Id: -2,Split Rating Amount: 0

Count = 20

2009-06-18  09:36:18,666   MAJOR com.vk.service.rating.server - Connec
+tion tmeout 

Count = 5
[download]

I want to store error count against the respective errors and display them. Can you please suggest\help on how to do this?

Thanks,

Ajay

Comment on Problem Parsing Log Files Select or Download Code

Replies are listed 'Best First'.
Re: Problem Parsing Log Files by jethro (Monsignor) on Jun 17, 2009 at 23:21 UTC
"I need to do it with two dimensional arrays". Why? Is it a school assignement? Otherwise a Hash (of Arrays) would make a lot more sense to find duplicate lines. The error message is the key, the timestamps are collected in the array. If you only need the last timestamp and the count, you can drop the array and store a counter and the last timestamp either as a concatenated string or a two-value array or hash	[reply]
Re^2: Problem Parsing Log Files by ajay.awachar (Acolyte) on Jun 18, 2009 at 00:27 UTC
Hi Jethro It's not an assignment. This script is for production logs. I don't just need a timestamp and count. I want time-stamp, error and count as a I mentioned in desired output. It's not necessery to do it with 2D array. As I need to store 3 components timestamp, error and count. I am not sure of what structure would be best to do this and how to do it. Thanks, Ajay	[reply]
Re^3: Problem Parsing Log Files by jethro (Monsignor) on Jun 18, 2009 at 00:43 UTC
If you use a hash of arrays as suggested above, the size of the array is your error count, the timestamps are stored in the array, and the error is the key of the hash. How to use a hash of arrays is described in the perl5 book for example, or in the manpage perldsc.	[reply]
Re^3: Problem Parsing Log Files by dsheroh (Monsignor) on Jun 18, 2009 at 10:40 UTC
I would go with HoA, given the description of the problem so far. The hash would use error messages as its keys, with references to two-element arrays (latest timestamp, count) as its values. Alternately, you might find it easier to just push timestamps onto the arrays. The size would then be the count and (assuming a sorted input file) the last-added timestamp would be the most recent.	[reply]