Re^8: search a large text file

very simple:

my $file_in_en=shift;
my $hash_en=to_hash($file_in_en);
print "@{$hash_en{'despite'}}";
[download]

Results:

expected result:

18
[download]

When I try to use a normal hash in this way I get a correct result:

my $file_in_en=shift;
my %hash_en=to_hash($file_in_en);
print "@{$hash_en{'despite'}}";

sub to_hash {
my %hash;
my $file = shift;
open(FILE, "<$file");
foreach $l (<FILE>) {
    my ($ngram,$line) = split /\t/, $l;
    push(@{ $hash{$ngram} }, $line);
    }
close FILE;
return %hash;
}
[download]

Comment on Re^8: search a large text file Select or Download Code

Replies are listed 'Best First'.
Re^9: search a large text file by jethro (Monsignor) on Feb 10, 2011 at 10:45 UTC
yes, this is what I talked about in "1)" above. Your hash is permanent, you have to fill it only once, ever. You need two scripts: This is the script you call once whenever your source file changes: `#!/usr/bin/perl #script 1. Fills the hash use warnings; use strict; use DBM::Deep; my $file_in_en=shift; my %hash_en=to_hash($file_in_en); sub to_hash { my $file = shift; my $db = DBM::Deep->new( "$file.db" ); $db->clear(); open(FILE, "<$file"); foreach $l (<FILE>) { my ($ngram,$line) = split /\t/, $l; push(@{ $db->{$ngram} }, $line); } close FILE; return $db; }` [download] And this is your search script: `#!/usr/bin/perl #script 2. Search item in hash use warnings; use strict; use DBM::Deep; my $file=shift; my $hash_en= DBM::Deep->new( "$file.db" ); print @{$hash_en->{'despite'}};` [download] UDPATE: corrected the search script	[reply] [d/l] [select]
Re^10: search a large text file by perl_lover_always (Acolyte) on Feb 10, 2011 at 13:47 UTC
Thanks, made it work with some small modifications! However the file still goes out of memory? which is strange! although I read the file line by line I get out of memory message using 10 gb of ram!	[reply]
Re^11: search a large text file by jethro (Monsignor) on Feb 10, 2011 at 15:37 UTC
Possibly this is because of too small pointers in the file (strangely the default is to limit the database size to 4GB). Read about that in section "LARGEFILE SUPPORT" of the DBM::Deep documentation. You also need a 64bit operating system to circumvent that limit If you don't have a 64bit system or still get out-of-memory errors it might be better to use a "real" database engine, like DBD::SQLite (which has the advantage that you don't have to install a separate database engine), mysql or postgreSQL (see erix answer).	[reply]