Keeping hash table in memory

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I have a very large hash table I don't want to reload everytime I set start up my program. I've got my program on a UNIX server, and I wanted to know if I could harness the power of using the environment variables to keep my hash table there.

Or better yet, just keep it in memory while another perl script accesses it. Point being I won't have to reload the table every time I have a new batch of items to query.

So I need to know, is that possible? If it is, any suggestions on how to do that would be great. Espeically useful would be info on how perl scripts may go about accessing those environment variables.

Thanks!

Comment on Keeping hash table in memory

Replies are listed 'Best First'.
Re: Keeping hash table in memory by moritz (Cardinal) on May 15, 2008 at 14:44 UTC
Environment variables are not designed for speed, and you can't set environment vars in a parent process, so forget that idea. I guess a good idea would be DBM::Deep and hope that your operating system's file cache keeps that in memory. It's quite fast actually, give it a try. Another idea might be memcached and one of its various perl bindings.	[reply]
Re: Keeping hash table in memory by dwm042 (Priest) on May 15, 2008 at 17:47 UTC
This kind of conversation has come up before ( e.g. making object hashes persistent), and one good (but abstract) suggestion is that you make a server to contain your information if you are wanting persistent information. In this context, a database is a server, as is perrin's use of Perl and a web server to deliver information to your application. David.	[reply]
Re: Keeping hash table in memory by mscharrer (Hermit) on May 15, 2008 at 15:55 UTC
As already posted before: forget environment variables for this application. The whole thing looks like to me that you should use a database - which can be accessed using a tied hash. If you don't want to use one you could do the following: Write a perl script which holds the hash and stays persistent in the background while allowing other perl scripts to query the hash elements using a unix socket or similar interface. You could even use a tied hash which automatically does the communication for you. Both introduces some runtime latency and you should carefully check if it isn't worse for your application then the load latency at start-up when you reload the hash.	[reply]
Re: Keeping hash table in memory by perrin (Chancellor) on May 15, 2008 at 16:31 UTC
Write your program to run under mod_perl and then access it by sending it HTTP requests from a command-line client. It will stay in memory and totally eliminate startup time.	[reply]
Re: Keeping hash table in memory by apl (Monsignor) on May 15, 2008 at 15:01 UTC
Will you be using the entire table each time? If not, consider storing the hash table in a temporary Sybase table.	[reply]
Re: Keeping hash table in memory by psini (Deacon) on May 15, 2008 at 15:51 UTC
I'm not sure if it what you need, but in a similar case I used IPC::ShareLite for defining a shared memory block. You can have a simple program to create the shared block, load it and then sleep; your main program can access the stored variable(s) as long as at least one process uses the shared block. Rule One: Do not act incautiously when confronting a little bald wrinkly smiling man.	[reply]
Re^2: Keeping hash table in memory by perrin (Chancellor) on May 15, 2008 at 16:32 UTC
IPC::ShareLite is pretty slow for large amounts of data. It runs the entire thing through Storable every time you read or write it.	[reply]
Re: Keeping hash table in memory by graff (Chancellor) on May 16, 2008 at 06:27 UTC
Have you looked at AnyDBM_File? You can choose to employ one of the particular DBM flavors described there, or you can just `use AnyDBM_File;` and apply whichever flavor was built in as the default for your perl installation. I posted a simple demonstration of the latter approach a few years back: Re: what is anydbm_file?. Nice thing about it: it's part of the perl core distro. HTH	[reply] [d/l]