use DBM::Deep;
tie my %total, 'DBM::Deep', $filename;
[download]

Now, you're limited to disk size, not RAM size.

Is there a way to hold the entire log contents itself in to some hash or array without utilising much memmory ,since the file size is too large

The node you replied to is the answer to your question. DBM::Deep is exactly the way to have a hash use disk instead of RAM.

---

My criteria for good software:

1. Does it work?
2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

BTW. You should probably be using exists in your conditions to prevent autovivifying keys that you don't use:

if( exists $total{"$source$dport"}{'from'} ) {
[download]

Without the exists, you will autovivify a key ("$source$dport") in %total that will have an anonymous hash assigned to it.

print exists $h{a} ? 1 : 0;;
0

print $h{a}{b} ? 1 : 0;;
0

print exists $h{a} ? 1 : 0;;
1

print $h{a};;
HASH(0x18eee74)
[download]

I don't think it will make much if any difference to your memory consumption in this case as you will go on to assign to the key anyway, but getting into the habit of using exists is a good thing in the long run as even empty anonymous hashes consume a fair amount of space.

If it is too big to fit in memory... then you have to use the disk! The easiest to use is probably GDBM_File, which will let you tie the hash to a disk file (but you will need to serialize the values of the hash). You could also go for a full DBMS, DBD::SQLite is very convenient, as the DB is a single file. it is pretty fast too.

In any case you will need to rewrite your code, and it will probably take much longer to run, there is no miracle here!

DBM::Deep will avoid both those problems. See my reply above.

---

My criteria for good software:

1. Does it work?
2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

I don't know if this is still the case but it used to be that when perl grew data structures for you, it would double the amount of memory it was using each time even if you really only needed just 1 more element. You could try to give your hash(es) a good number of buckets to start with by assigning to keys thusly:

keys %total = 500;
[download]

I think the following would be equivalent to the OP code in terms of what it does, but might take less memory and might run a bit faster:

while (<LOG>) {
    my ($source, $sport, $to, $dport, $proto, $packs, $bytes) = split;
    my $key = "$source$dport";
    if ( exists( $total{$key}{from} )) {
        $total{$key}{connects}++;
        $total{$key}{bytes} += $bytes;
    }
    else {
        $total{$key} = { from => $source,
                         to => "$to:$dport",
                         bytes => $bytes,
                       };  # maybe should set 'connects => 1' as well?
    }
    $total += $bytes;
}
[download]

Here are a few (potentially meaningless) benchmarks about the trade-off between more top-level (simple, flat) hashes vs. a single top-level hash with more sub-hash keys (I put a "sleep" in there so I could study the memory/time consumption once the hashes were filled):

perl -e '$k="aaaaa";
 for $i (1..1_000_000)
 { $h1{$k}={foo=>"bar",bar=>"foo",iter=>$i}; $h1{$k}{total}++; $k++}
 sleep 20'
## consumes 344 MB in ~14.4 sec

perl -e '$k="aaaaa";
 for $i (1..1_000_000)
 {$h1{$k}={foo=>"bar",bar=>"foo",iter=>$i}; $h2{$k}++; $k++}
 sleep 20'
## consumes 352 MB in ~15.0 sec

perl -e '$k="aaaaa";
 for $i (1..1_000_000)
 {$h1{$k}={foo=>"bar",bar=>"foo"}; $h2{$k}++; $h3{$k}=$i; $k++}
 sleep 20'
## consumes 360 MB in ~16.5 sec
[download]

The first thing you can do is try to rewrite your algorithm to process the input as sequentially as possible, I can not help you here too much because I don't fully understand what you want to get as the final result of the processing, but if you would tell us about it...

You can also store the data on the hash packed using vec, both for the keys and the values. The key will consume 6 bytes and the value 10 bytes plus the scalar (SV) overhead (you don't need to store $source both in the key and on the value). That would probably reduce your memory requirements to 1/10.

Regarding the code you have posted, making the keys as "$source$dport" is ambiguos, for instance 10.0.1.10.1445 could be ....1, 445; ....14, 45; ....144, 5. Better you use $total{$source,$dport} that is equivalent to $total{"$source\0$dport"}.

Finally, you are using as keys only the source IP and the destination port for %total and storing the target IP. That doesn't make sense to me as generally, there could be different target IPs. For instance, an user (source IP) browsing the web (port 80) would be accessing several servers (different target IPs).