It sounds like you are running out of memory.

A 1_000_000 key hash with a reference to an empty hash as the value requires around 160 MB. For 8_000_000 keys, that would be 1.2 GB. Even with duplicates at each level, that is still a lot of hashes.

How much memory do you have in your machine? Have you monitored the memory consuption as the program runs?

The machine is an SGI Origin 16 processor HPC with 1 gig ram per processor. When running job accounting, nothing of note seems to happen. I understand a database is desired, and I intend to go that route eventually, but I would like to have both options - database and text files manipulation.

Nice hardware, but ... 1 GB/Processor? I know nothing of that hardware, but that (again) sounds like any given process will be limited to 1 GB minus any OS overhead. It very much depends upon the distribution of the contents of the file, but I could quite see the 8 million lines building a hash structure > 1 GB.

Most times when Perl run's out of ram on my machine I get Perl's "Out of memory" error, but occasionally I get a segfault.

Maybe your job accounting would be telling you if memory was a problem--I've not the vaguest clue what that might contain--and you can rule it out, but if you have access to a top-like live monitoring program, it would be worth checking it out.

I think I would try filtering the input file into smaller files, say by protocol (assuming there not all icmp) and then process those separately and combine the results.

You don't actually show what your doing with that monster structure. It looks like your just counting the number of dropped packets per

date/proto/dst:port/src:port.

If that is all your are doing, then there little point in building the deep structure. You would achieve the same results by concatenating all those values into a string and using a single level of hash.

That said, if the problem is memory, that may not help much.

---

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

I agree with the others that you are running out of memory. I would note that the top level key "$month.year" appears redundant as it is a constant so you could loose that level although that should not make any significant difference. For the specific task you show (counting) you don't need a hash of hash of hash.....just to increment a counter. You could just stringify the key:

my $key = join '|', "$month.year", $fmt_proto, $fmt_dest_ip, $fmt_dest
+_port, $fmt_src_ip, $fmt_src_port;
$mon_log{$key} += $fmt_drp_packets;
[download]

This removes all those expensive levels of keys but still gets you your count. You can breakdown the key with split as required. This may use less memory even though the keys are longer and contain redundant data. Memory consumption will depend on how many keys you end up with. It is a crappy way to do it compared to a database. It looks like tab sep data (or it could be) so you could do in MySQL something like:

create table stuff (
    proto char(4),
    src_ip char(15),
    etc....
    drp_packets int,
    index(proto),
    index(src_ip),
    etc....
)

load data local infile '/blah/blah.dat' into table stuff

select sum(packets) where src_ip = 1.2.3.4 and .....
[download]

You can then make any queries you want.....

What you are actually doing could be handled in a different way. If you sort the input file (unix sort will handle it fine and do it fastest) then you can simply iterate over to generate your counts using a line merge strategy every time you find a new proto/src/dest/port combo:

my $current_rec = '';
my $current_count = 0;
my($proto, $dest_ip, $src_ip, $dest_port, $src_port, $drp_packets, $co
+untry, $rec);

while(<REPORTFILE>){
    ($proto, $dest_ip, $src_ip, $dest_port, $src_port, $drp_packets, $
+country) = split' ';
    $rec = join "\t", $proto, $dest_ip, $src_ip, $dest_port, $src_port
+;
    if ( $rec eq $current_rec ) {
        $current_count += $drp_packets;
    }
    else {
        print OUTFILE $current_rec, "\t", $current_count;
        $current_rec = $rec;
        $current_count = $drp_packets;
    }
}

# now print any hanging rec
print OUTFILE $current_rec, "\t", $current_count if $rec eq $current_r
+ec;
[download]

This will also be probably an order of magnitude or two faster than using a hash. You can do a sort -nrk6 outfile > drop_sort to sort by dropped packets. BTW your split will split 'United States' into two tokens. use split ' ', $_, 7 to get what you expect.

-sam