The premise that you need to store millions of items in memory is flawed. No amount of optimization is going to save you when you need such a huge amount of memory.

()-()
 \"/
  `                                                     

And to further expand on ignatz, you can switch to something like DB_File (which is in core perl) or BerkeleyDB and not have to change your code much. You keep your hash - it's just tied to a database. I've used BerkeleyDB to good effect for working with large datasets that didn't fit into memory.

Update: you'll probably also want to use something to store your structures as well: Storeable, MLDBM, FreezeThaw, Data::Dumper, etc. You can't just store raw structures but the idea is the same.

__SIG__
printf "You are here %08x\n", unpack "L!", unpack "P4", pack "L!", B::
+svref_2object(sub{})->OUTSIDE
[download]

You don't say which part runs out of memory, the load or the display.

In any case, it appears that you can divide and conquer .. try running the script on smaller chunks, then add the intermediate results together to get a final result.

Thanks Alex, I was thinking about smaller chunks or maybe using 3 arrays. I parse all of this stuff and then it is loaded into a database.

while (<IN>) {
    my ($key, $x, $y) = split;
    my @from_cache = split ' ', $cache{$key};
    $from_cache[0] += $x;
    $from_cache[1] += $y;
    $cache{$key} = join ' ', @from_cache;
}
[download]

thanks to everyone. I will give this a try.

If you're curious as to the current size of things, you can always play with the Devel::Size module, which'll figure out how much memory your hash is using right now.

why bother trying to store "a huge amount of data" in memory, when all you are doing is processing it linearly?

Sort your huge data file (by the key) first, and then process it with something like this...

#!/usr/local/bin/perl

use warnings;
use strict;

my ($key, $first, $second) = split ' ', <>;

while (<>) {
    my @line = split;
    if ($key eq $line[0]) {
    $first += $line[1];
    $second += $line[2];
    } else {
    print "$key $first $second\n";
    ($key, $first, $second) = @line;
    }
}

print "$key $first $second\n";
[download]

(something like "sort your/big/ass/data/file.txt | monk.pl")

perlfaq3 offers this advice concerning using less memory.

How can I make my Perl program take less memory?

In some cases, using substr() or vec() to simulate arrays can be highl
+y beneficial. For example, an array of a thousand booleans will take 
+at least 20,000 bytes of space, but it can be turned into one 125-byt
+e bit vector--a considerable memory savings. The standard Tie::Substr
+Hash module can also help for certain types of data structure. If you
+'re working with specialist data structures (matrices, for instance) 
+modules that implement these in C may use less memory than equivalent
+ Perl modules.
[download]

I did some playing around with Tie::SubstrHash a while ago, but my results were inconclusive for my purposes, but depending upon your data, it might prove more frugal than using a standard hash.

I was going to suggest CB_File, but i see diotalevi already said that.

<silly>
I did notice that the following was missing :

use less 'memory';
[download]