Best way to pick apart log file.

dru145 has asked for the wisdom of the Perl Monks concerning the following question:

Hey Monks,

I'm in need of some assistance once again. I have a log file in this format (*note: ip's have been changed to protect the guilty):

1;30Nov2001;17:08:25;192.148.14.2;log;accept;;hme0;outbound;udp;192.14
+8.84.4;24.248.34.99;domain-udp;1103;43;85;;;;;;;;;;;;;;;
2;30Nov2001;17:08:25;192.148.14.2;log;drop;;hme0;inbound;tcp;43.228.94
+.254;192.148.11.47;netbios-ssn;18803;48;89;;;;;;;;;;;;;;;
3;30Nov2001;17:08:26;192.148.14.2;log;drop;;hme0;inbound;tcp;45.93.220
+.223;192.148.24.139;auth;1323;40;89;;;;;;;;;;;;;;;
4;30Nov2001;17:08:26;192.148.14.2;log;drop;;hme0;inbound;tcp;45.93.220
+.223;192.148.24.139;auth;1323;40;89;;;;;;;;;;;;;;;
5;30Nov2001;17:08:26;192.148.14.2;log;accept;;qfe2;inbound;tcp;192.148
+.84.144;24.248.34.97;http;4719;44;85;;;;;;;;;;;;;;;
6;30Nov2001;17:08:26;192.148.14.2;log;accept;;hme0;outbound;tcp;192.14
+8.84.144;24.248.34.97;http;4719;44;85;;;;;;;;;;;;;;;
7;30Nov2001;17:08:26;192.148.14.2;log;accept;;qfe2;inbound;tcp;192.148
+.84.144;24.248.34.97;http;4721;44;85;;;;;;;;;;;;;;;
8;30Nov2001;17:08:26;192.148.14.2;log;accept;;hme0;outbound;tcp;192.14
+8.84.144;24.248.34.97;http;4721;44;85;;;;;;;;;;;;;;;
8;30Nov2001;17:08:26;192.148.14.2;log;accept;;hme0;outbound;tcp;192.14
+8.84.144;24.248.34.97;http;4721;44;85;;;;;;;;;;;;;;;
9;30Nov2001;17:08:26;192.148.14.2;log;accept;;qfe2;inbound;tcp;192.148
+.27.154;205.188.145.185;http;4394;44;85;;;;;;;;;;;;;;;
10;30Nov2001;17:08:26;192.148.14.2;log;accept;;hme0;outbound;tcp;192.1
+48.27.154;205.188.145.185;http;4394;44;85;;;;;;;;;;;;;;;
11;30Nov2001;17:08:26;192.148.14.2;log;accept;;qfe2;inbound;tcp;192.14
+8.27.154;205.188.145.185;http;4397;44;85;;;;;;;;;;;;;;;
12;30Nov2001;17:08:26;192.148.14.2;log;accept;;hme0;outbound;tcp;192.1
+48.27.154;205.188.145.185;http;4397;44;85;;;;;;;;;;;;;;;
[download]

What I am trying to do is check and see if an ip address along with a service (http) remains constant for a certain number of entries (we'll say 5 for this example). I need the script to be able to determine if ANY service shows up 5 times with the ip address given. Here is what I've come up with:

#!/usr/bin/perl -w 
use strict;
                         
my $log = './log';                                                    
+    
my ($count1, $dst, $service, %count, %hash);
                     
my $ip = '24.248.36.97'; 
    
open (LOG, $log) or die "Can't open $log: $!"; 
while (<LOG>){ 
                     
   ($dst, $service) = (split /;/)[11, 12];
   $count1++ if ($dst =~ /$ip/);
 
   %hash = (service => $service); 
   foreach my $key ($hash{service}){
     $count{$key}++;   
   }                   
}                      
  
foreach my $key1 (keys %count){ 
 print "There are at least 5 occurences of $key1 and $ip.\n" if ($coun
+t{$key1} >= 5 && $count1 >= 5);
}
[download]

This works, but when I run it with -w and strict, I get the following output:

Use of uninitialized value in pattern match (m//) at ./misconfig.pl li
+ne 13, <LOG> line 36.
Use of uninitialized value in hash element at ./misconfig.pl line 17, 
+<LOG> line 36.
There are at least 5 occurences of http and 24.248.36.97.
[download]

I have a few questions if somebody would be most gratefull to help me out:

Is the best method of doing this?
Why am I getting these warnings?

Thanks in advance,
Dru

Comment on Best way to pick apart log file. Select or Download Code

Replies are listed 'Best First'.
Re: Best way to pick apart log file. by mkmcconn (Chaplain) on Dec 27, 2001 at 22:45 UTC
Try inserting into your while loop: `next if m/^\s$/;` [download] It appears your code is failing because there are blank lines, or lines not delimited with ';'. Is that true? mkmcconn By the way, I really admire that line: `($dst, $service) = (split /;/)[11, 12];`*	[reply] [d/l] [select]
Re: Re: Best way to pick apart log file. by Rich36 (Chaplain) on Dec 27, 2001 at 23:00 UTC
That probably is the case. I ran this with trailing blank lines and got the same error. When the blank lines are removed, the error goes away - even using strict and warnings. The other thing that got rid of another error was to actually define `$count1` as `$count1 = 0`. Otherwise I got an error `Use of uninitialized value in numeric ge (>=) at logtest.pl line 47` at the end of the code. Rich36 There's more than one way to screw it up...	[reply] [d/l] [select]
Re: Re: Best way to pick apart log file. by dru145 (Friar) on Dec 27, 2001 at 23:21 UTC
Yes, that is true, there are blank lines in the logs. I added your next statement and it worked like a champ. By the way, I really admire that line: `($dst, $service) = (split /;/)[11, 12];` Thanks, but I can't take credit for it. It was suggested by a most excellent monk, Juerd, in this post Another Array Problem: comparing.. -Dru Another satisfied monk	[reply] [d/l]