Re: Re: A memory efficient hash, trading off speed

A superlative idea... However, I didn't make my intentions apparent enough - my fault.
At the moment the system uses such a large amount of memory that portions of the data set are paged off to disk. This slowdown (While, admittedly, being somewhat slower than simply pulling everything from disk, there is a comparison to be drawn with disk based data access) makes the project unuseable due to having to pull data off disk.

I was willing to trade off speed in adding data to the hash, but not in doing an actual hash lookup.
What I'm asking is quite likely unreasonable, but then again, thats why I put it to all you fine and smart folks - to show me where the problems in my plans lay.

JP,
-- Alexander Widdlemouse undid his bellybutton and his bum dropped off --

Comment on Re: Re: A memory efficient hash, trading off speed - does it already exist?

Replies are listed 'Best First'.
Re: Re: Re: A memory efficient hash, trading off speed - does it already exist? by perrin (Chancellor) on Feb 04, 2003 at 04:49 UTC
You could try DB_File with it's in-memory option. It will store the keys as C strings, so they may be more compact.	[reply]
Re: Re: Re: A memory efficient hash, trading off speed - does it already exist? by Tardis (Pilgrim) on Feb 04, 2003 at 23:34 UTC
Have you actually tried tieing your hashes to a DBM file? You may find the speed hit is not anywhere near as bad as you think. I have DBM files with many millions of keys, and fetches are still lightning fast. Considering it's probably about 4 lines worth of code to try, give it a shot :-) I don't think it's even worth trying to draw conclusions about how fast a DBM file access will be by looking at the slowdown of a swapping process. Apples and Oranges.	[reply]