Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: Re: A memory efficient hash, trading off speed - does it already exist?

by JPaul (Hermit)
on Feb 04, 2003 at 04:13 UTC ( #232458=note: print w/replies, xml ) Need Help??


in reply to Re: A memory efficient hash, trading off speed - does it already exist?
in thread A memory efficient hash, trading off speed - does it already exist?

A superlative idea... However, I didn't make my intentions apparent enough - my fault.
At the moment the system uses such a large amount of memory that portions of the data set are paged off to disk. This slowdown (While, admittedly, being somewhat slower than simply pulling everything from disk, there is a comparison to be drawn with disk based data access) makes the project unuseable due to having to pull data off disk.

I was willing to trade off speed in adding data to the hash, but not in doing an actual hash lookup.
What I'm asking is quite likely unreasonable, but then again, thats why I put it to all you fine and smart folks - to show me where the problems in my plans lay.

JP,
-- Alexander Widdlemouse undid his bellybutton and his bum dropped off --

  • Comment on Re: Re: A memory efficient hash, trading off speed - does it already exist?

Replies are listed 'Best First'.
Re: Re: Re: A memory efficient hash, trading off speed - does it already exist?
by perrin (Chancellor) on Feb 04, 2003 at 04:49 UTC
    You could try DB_File with it's in-memory option. It will store the keys as C strings, so they may be more compact.
Re: Re: Re: A memory efficient hash, trading off speed - does it already exist?
by Tardis (Pilgrim) on Feb 04, 2003 at 23:34 UTC
    Have you actually tried tieing your hashes to a DBM file? You may find the speed hit is not anywhere near as bad as you think.

    I have DBM files with many millions of keys, and fetches are still lightning fast.

    Considering it's probably about 4 lines worth of code to try, give it a shot :-)

    I don't think it's even worth trying to draw conclusions about how fast a DBM file access will be by looking at the slowdown of a swapping process. Apples and Oranges.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://232458]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (1)
As of 2023-05-31 02:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?