You can trade time spent to load the hash into memory against time spent accessing an element by using a tied hash like DB_File, or, if your hash is basically constant, CDB_File. But that will slow down every access, so you will have to benchmark whether all your hash accesses will sum up to the 7 seconds load time. You could also investigate whether simply storing the hash as Perl code via Data::Dumper and loading it via do will be faster (I doubt it), or whether using/writing a daemon like Memcached improves things.

Sidestep the problem. Perhaps you should look into leaving the hash on disk (e.g. use BerkeleyDB, or possibly step up to a full RDBMS and DBI) rather than loading it into memory?

The only real way to speed up the load is to load less data at a time. You could go the route of breaking the files by the first letter (or paior of letters) as I suggested in Re: reading dictionary file -> morphological analyser. except this time using Storable.

That said, using Storable doesn't buy you much in performance for text data in a single level hash. The memory still has to be allocated and the hash built.

$dict = retrieve("hash.txt");
[download]

Dave.

Just in case you want to stick with the hash approach (the DB approach and the other suggestions would be fine, too), you could also speed up things by avoiding to reload the hash every time. That's essentially the client/server approach I was hinting at in Re: reading dictionary file -> morphological analyser. I.e., you split the program into a server and a client. The server loads the hash once and keeps running, providing a 'dict' service via some socket. The client just connects to the socket, sends the request (the word to be looked up), and reads the server's reply.

Here's a minimal example, just to get you started. (This is not production quality code, and could be improved in many ways (e.g. by making a proper daemon out of it, better error handling, etc.), but I wanted to keep it simple...)

The server:

use IO::Socket;
use Storable;

my $dict = retrieve("hash.txt");

my $sock = IO::Socket::INET->new(
               LocalAddr => "localhost:8888",
                                     # any available port you like
               ReuseAddr => 1,
               Listen    => 2,
           ) or die "$0: can't create listening socket: $!\n";

while (1) {
    my $conn = $sock->accept();  # wait for connection

    next unless ref $conn;       # (just in case...)

    my $query = <$conn>;
    chomp $query;

    print STDERR "query: $query\n";   # just for debugging

    my $reply = (exists $dict->{$query}) ? "FOUND\n" : "NOT FOUND\n";
    print $conn $reply;

    close $conn;
}
[download]

The client:

use IO::Socket;

sub connect_server {
    my $sock = IO::Socket::INET->new(
                   PeerAddr  => "localhost:8888",  
               ) or die "$0: can't connect: $!\n";
    return $sock;
}

my @inputs = qw(
foo
fooed
fooen
);

for my $input (@inputs) {
    
    my $conn = connect_server();
    
    print $conn "$input\n";       # send the query
    my $reply = <$conn>;          # read the response

    close $conn;

    print "found '$input' in lexicon\n" if $reply eq "FOUND\n";
}
[download]

Even though this code is opening a new connection for every word being looked up, it's still pretty fast (around 2 ms per lookup, on my machine).

(To play with the example, start the server in one terminal (as I mentioned, it's not a daemon, so it will stay in the foreground), and then run the client from another terminal... )

db_load -c duplicates=1 -T -t hash -f dict.txt dict.db
[download]