Basically I'm using the data structure to run comparisons of two lists of bags. For each item in list a, for each item in list b, there are about 7 bags of words. I need to compare all the bags of each item in list a to all the bags of each item in list b, using operations like "intersect."

That data structure is going to take up about 11.7 Gigabytes of memory. And that's with the degenerate case you give where each HoHoH has a single digit integer as its data. (Reduce the outer loop to 10 so it will fit into memory, put in a busy/wait loop so you have time to look up the memory usage before it exits, then look in the process manager to get the memory usage.)

use strict;
use warnings;

my %hash = ();
foreach my $a (1..10) {
    foreach my $b (1..4000) {
        foreach my $c (1..10) {
            foreach my $d (1..7) {
                $hash{$a}{$b}{$c}{$d} = $d;
            }
        }
    }
    print "$a\n";
}
print "here\n";

next while (!<STDIN>);
[download]

I come up with about 247000KB used. Take out about 2K for the perl interpreter overhead, multiply by 50, divide by 1024²... 11.7 GB.

Even converting to arrays, it takes up about 5 Gigabytes.

It would seem that either you are going to have to rework your algorithm or use a database. I'm not clear what exactly you are attempting to accomplish.

It would seem that either you are going to have to rework your algorithm or use a database.

Or use a language with less overhead. C or C++ come to mind. Or buy more memory. You should be able to get 12GB of RAM for about $1200 (with 4GB memory sticks). I paid more than that for a 486 with 4MB back in the day.

That sounds like a database job to me.

---

Caution: Contents may have been coded under pressure.

I need to compare all the bags of each item in list a to all the bags of each item in list b, using operations like "intersect."

Somehow, that doesn't seem to explain why you have four nested dimensions in a hash structure (or in an array structure, which would seem more appropriate based on what you've told us).

Maybe you can decompose the problem so that you use hashes or arrays of just two dimensions? Just a thought...