How store a hash table on disk and work with it

ReneB has asked for the wisdom of the Perl Monks concerning the following question:

Hi, i am loading 2x3M tuples into two hash tables and have to compare them after some rulez. Problem is that the size of the hash tables in memory are about 600MB. Since the machine i have to run the program on has only 512MB installed. i was thinking about storing the stuff on disk and work from there. Can you tell me how this would be possible

            my $recordsize=160;
            my ( $portingID, $cli );
            
            print "${INFO}building hash table\t\t - '$infile'\n";
            until( eof(IN) ){
                read(IN, my $data, $recordsize) == $recordsize
                $cli      =substr($data, 0,9);
                $portingID=substr($data,30,6);
    
                $old_tab{ $cli } = $portingID if $filenum==1;
                $new_tab{ $cli } = $portingID if $filenum==2;
                $actld++;
             }
            close IN;
[download]

Comment on How store a hash table on disk and work with it Download Code

Replies are listed 'Best First'.
Re: How store a hash table on disk and work with it by Corion (Patriarch) on Dec 04, 2008 at 08:21 UTC
See tie and especially DB_File or DBM_File, or whatever other btree storage is hopefully installed on your Perl.	[reply]
Re^2: How store a hash table on disk and work with it by ReneB (Initiate) on Dec 04, 2008 at 13:20 UTC
many thanx, that was very helpful, will try to PPM that module.	[reply]
Re: How store a hash table on disk and work with it by gone2015 (Deacon) on Dec 04, 2008 at 09:39 UTC
I note that you have: `$old_tab{ $cli } = $portingID if $filenum==1; $new_tab{ $cli } = $portingID if $filenum==2;` [download] You'd save memory by packing the old and new `$portingID` together and using just one hash table. Shouldn't take long to try it and see if it's enough of a saving.	[reply] [d/l] [select]
Re^2: How store a hash table on disk and work with it by ReneB (Initiate) on Dec 04, 2008 at 13:00 UTC
i have to compare the two has tables after some rules which are in the rest of the program, so thanx but i think no way to store all that stuff in one hash.	[reply]
Re: How store a hash table on disk and work with it by dragonchild (Archbishop) on Dec 05, 2008 at 12:49 UTC
This is exactly why DBM::Deep was written. My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
A reply falls below the community's threshold of quality. You may see it by logging in.