Re^2: a large text file into hash

Good observations and thanks! In fact I need it once to be created and then I will access it many times. so this one time processing for me takes lots of time however it's important to be accessed later easily and not very time consuming. I can have a big memory of 50 GB but still is not enough and it goes out of memory! I tried to create the hash and then tie it, it would not also possible for me or maybe I do some errors that it still goes out of memory!

my $t = tie(%hash, Tie::IxHash);
foreach my $line (@file){
    $line_count++;
    my @ngrams=produce_ngrams($line);
    foreach my $ngram (@ngrams) {
        #$t->Push(@{ $hash{$ngram} } => $line_count); 
        push(@{ $hash{$ngram} }, $line_count);
    }
}
[download]

I have also no idea if I tie the hash, later on how can I access it from my hard drive.

Comment on Re^2: a large text file into hash Download Code

Replies are listed 'Best First'.
Re^3: a large text file into hash by BrowserUk (Patriarch) on Jan 28, 2011 at 14:01 UTC
I tried to create the hash and then tie it, Tie::IxHash a) doesn't store to disk; b) use 2 or 3 time as much memory as a standard hash. It's purpose is to remember the order in which the keys of the hash were added which is unnecessary for your use. You should not be using this module. If you are going the tie'd hash root, then you need to use a module that ties the hash to a disk file. Previously I'd have recommended BerkeleyDB, but since Oracle grabbed Sun, you have to sign up and agree to let them do whatever they want before they'll let you download anything. There are alternatives but I don't have much experience of them, so I cannot make a recommendation. But, if you have 50GB of ram available, then you ought to be able to hash your 1 GB file in memory with ease. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: a large text file into hash by perl_lover_always (Acolyte) on Jan 28, 2011 at 14:12 UTC
well, since I make the hash once and I want to use it several times later, then I'd prefer to keep it in the hard disk for the later access. what is your suggestion in this scenario?	[reply]
Re^5: a large text file into hash by BrowserUk (Patriarch) on Jan 28, 2011 at 14:48 UTC
As I said, BerkeleyDB is the most recommended solution for disk-based hashes. Alternatives include MLDBM, SQLite_file, DBM::Deep etc. Or you could use an RDBMS (Postgres/MySQL/other). Perhaps via Tie::DBI. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]