Looping Headaches

ACiD has asked for the wisdom of the Perl Monks concerning the following question:

Hello to all. I am stuck trying to do something primative (for y'all at least) The following would be an example of the input file:

2003Aug01/
EVENT1: [HEARTBEAT] (IP=1260953,ICMP=898,TCP=1236959,UDP=22477,EVENTS=
+950,DROP=0,VER=6.0.1)
EVENT1: [HEARTBEAT] (IP=1217149,ICMP=874,TCP=1193416,UDP=22133,EVENTS=
+811,DROP=0,VER=6.0.1)
2003Aug02/
EVENT1: [HEARTBEAT] (IP=640626,ICMP=855,TCP=620893,UDP=18614,EVENTS=71
+4,DROP=0,VER=6.0.1)
EVENT1: [HEARTBEAT] (IP=652513,ICMP=830,TCP=631758,UDP=19671,EVENTS=72
+2,DROP=0,VER=6.0.1)
[download]

Basically, from my code below, I read a new line $line, if its a date, grab and put into date variable, otherwise, process (count and total) the data lines until a date line pops up again, then print the total for the day in an out file. I have ripped this code apart several time until my hair is in a pile on the floor.

#!/usr/bin/perl -w -d
use strict;
my $date;
my $line;
my ($tot_ip_cnt,$tot_icmp_cnt,$tot_tcp_cnt,$tot_udp_cnt,$tot_events,$t
+ot_drops) = '0';

open (INFILE, "< hbeat-app.log.samp") or die ("Error Opening File $!\n
+");   
open (OUTFILE, ">> HB_APP_totals.log.CSV") or die ("Error Opening File
+ $!\n");
print OUTFILE "date,ip_cnt,icmp_cnt,tcp_cnt,udp_cnt,events,drops \n";

while ( my $line = <INFILE>) {
   if ($line =~ m/^(\d{4})(\w{3})(\d+).+/) {
     $date = join ' ',($2,$3,$1);
     next;
   } else {
   #  while ($line =~ m/^EVENT1\:\s+\[\w+\]\s+\(IP\=(\d+)\,ICMP\=(\d+)
+\,TCP\=(\d+)\,UDP\=(\d+)\,EVENTS\=(\d+)\,DROP\=(\d+)\,.+/) {
     until ($line =~ m/^(\d{4})(\w{3})(\d+).+/) {
       $tot_ip_cnt += $1;
       $tot_icmp_cnt += $2;
       $tot_tcp_cnt += $3;
       $tot_udp_cnt += $4;
       $tot_events += $5;
       $tot_drops += $6;
       print "$line \n";
       next;
       print "$line \n";
       }
     }
   print OUTFILE "$date,$tot_ip_cnt,$tot_icmp_cnt,$tot_tcp_cnt,$tot_ud
+p_cnt,$tot_events,$tot_drops\n";
}
close OUTFILE;
close INFILE;
[download]

With this and other modifications I can get parts to work, but not the whole thing (notice the while/until parts commented above). Any help is GREATLY appreciated !!!

Comment on Looping Headaches Select or Download Code

Replies are listed 'Best First'.
Re: Looping Headaches by tedrek (Pilgrim) on Aug 16, 2003 at 17:35 UTC
This appears to give sane output for the test data :). #!/usr/bin/perl -w use strict; my @fields = qw/ip icmp tcp udp events drops/; my %count; my $date; open (INFILE, "<&","DATA") or die ("Error Opening File $!\n"); open (OUTFILE, ">&", 'STDOUT') or die ("Error Opening File $!\n"); print OUTFILE "date,ip_cnt,icmp_cnt,tcp_cnt,udp_cnt,events,drops \n"; while ( my $line = <INFILE>) { if ($line =~ m/^(\d{4})(\w{3})(\d+).+/) { print OUTFILE join(",", $date, @count{@fields}), "\n" if defin +ed $date; %count = (); # reset the counters $date = join ' ',($2,$3,$1); next; } elsif ($line =~ m/^EVENT1:\s+\[\w+\]\s+\(IP=(\d+), ICMP=(\d+), TCP=(\d+), UDP=(\d+), EVENTS=(\d+), DROP=(\d+),.+/x) { $count{ip} += $1; $count{icmp} += $2; $count{tcp} += $3; $count{udp} += $4; $count{events} += $5; $count{drops} += $6; #print "$line \n"; next; } else { # Line that is neither a date or a event next; } } print OUTFILE join(",", $date, @count{@fields}), "\n" if defined $date +; __DATA__ 2003Aug01/ EVENT1: [HEARTBEAT] (IP=1260953,ICMP=898,TCP=1236959,UDP=22477,EVENTS= +950,DROP=0,VER=6.0.1) EVENT1: [HEARTBEAT] (IP=1217149,ICMP=874,TCP=1193416,UDP=22133,EVENTS= +811,DROP=0,VER=6.0.1) 2003Aug02/ EVENT1: [HEARTBEAT] (IP=640626,ICMP=855,TCP=620893,UDP=18614,EVENTS=71 +4,DROP=0,VER=6.0.1) EVENT1: [HEARTBEAT] (IP=652513,ICMP=830,TCP=631758,UDP=19671,EVENTS=72 +2,DROP=0,VER=6.0.1) [download]	[reply] [d/l]
Re: Looping Headaches by tcf22 (Priest) on Aug 16, 2003 at 17:02 UTC
I don't think that you want to print this out after every loop. I think you want to print out after every day. Maybe try this. Also I think the way you are looping through the lines is a little messed up. The while is looping through the lines, and the until really isn't doing anything. Try something like this(untested): #!/usr/bin/perl -w -d use strict; my $date; my $line; my ($tot_ip_cnt,$tot_icmp_cnt,$tot_tcp_cnt,$tot_udp_cnt,$tot_events,$t +ot_drops) = (0,0,0,0,0,0); open (INFILE, "< hbeat-app.log.samp") or die ("Error Opening File $!\n +"); open (OUTFILE, ">> HB_APP_totals.log.CSV") or die ("Error Opening File + $!\n"); print OUTFILE "date,ip_cnt,icmp_cnt,tcp_cnt,udp_cnt,events,drops \n"; while ( my $line = <INFILE>) { if ($line =~ m/^(\d{4})(\w{3})(\d+).+/) { print OUTFILE "$date,$tot_ip_cnt,$tot_icmp_cnt,$tot_tcp_cnt,$t +ot_udp_cnt,$tot_events,$tot_drops\n" if(length($date) > 0); $date = join ' ',($2,$3,$1); } else { $line =~ m/^EVENT1\:\s+\[\w+\]\s+\(IP\=(\d+)\,ICMP\=(\d+)\,TCP\=( +\d+)\,UDP\=(\d+)\,EVENTS\=(\d+)\,DROP\=(\d+)\,.+/; $tot_ip_cnt += $1; $tot_icmp_cnt += $2; $tot_tcp_cnt += $3; $tot_udp_cnt += $4; $tot_events += $5; $tot_drops += $6; } } print OUTFILE "$date,$tot_ip_cnt,$tot_icmp_cnt,$tot_tcp_cnt,$tot_udp_c +nt,$tot_events,$tot_drops\n"; close OUTFILE; close INFILE; [download] Update: Added print after loop to print out last day.	[reply] [d/l]
Re: Looping Headaches by ACiD (Novice) on Aug 16, 2003 at 20:55 UTC
Thank you very much. It worked after I `undef`'ed the counters after the date. Again. thanks. Perl and its monastery rocks !!	[reply] [d/l]
Re: Looping Headaches by Not_a_Number (Prior) on Aug 18, 2003 at 18:02 UTC
Too late, I fear, but here is one way way to do what you want that is a lot more condensed and uses far fewer variables (it's also IMHO more scaleable to similar problems): my @to_match = qw ( IP ICMP TCP UDP EVENTS DROP ); my $pat = join ',', map { $_ . '=(\d+)' } @to_match; my ( $date, @total_counts ); while ( <DATA> ) { if ( /^(\d{4})(\w{3})(\d{2})/ ) { # print OUTFILE join ( ',', "$date", @total_counts ), "\n" if $d +ate; print join ( ',', "$date", @total_counts ), "\n" if $date; $date = "$2 $3 $1"; @total_counts = (); } elsif ( /^EVENT1:\s+\[\w+\]\s+/ ) { if ( my @counts = /$pat/ ) { $total_counts[$_] += $counts[$_] for 0 .. $#counts; } else { print STDERR "Possible corrupt data line $.:\n $_", "Line ignored!\n"; } } } # print OUTFILE join ( ',', "$date", @total_counts ) if $date; print join ( ',', "$date", @total_counts ) if $date; __DATA__ 2003Aug01/ EVENT1: [HEARTBEAT] (IP=1260953,ICMP=898,TCP=1236959,UDP=22477,EVENTS= +950,DROP=0,VER=6.0.1) EVENT1: [HEARTBEAT] (IP=1217149,ICMP=874,TCP=1193416,UDP=22133,EVENTS= +811,DROP=0,VER=6.0.1) 2003Aug02/ EVENT1: [HEARTBEAT] (IP=640626,ICMP=855,TCP=620893,UDP=18614,EVENTS=71 +<oops!> Connection down! Reconnecting....... EVENT1: [HEARTBEAT] (IP=640626,ICMP=855,TCP=620893,UDP=18614,EVENTS=71 +4,DROP=0,VER=6.0.1) EVENT1: [HEARTBEAT] (IP=652513,ICMP=830,TCP=631758,UDP=19671,EVENTS=72 +2,DROP=0,VER=6.0.1) [download] HTMHH (Hope That Might Have Helped :) dave	[reply] [d/l]