Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: A memory efficient hash, trading off speed - does it already exist?

by BrowserUk (Patriarch)
on Feb 03, 2003 at 23:13 UTC ( [id://232405]=note: print w/replies, xml ) Need Help??


in reply to A memory efficient hash, trading off speed - does it already exist?

Take a look at Tie::SubstrHash. I think that it's a part of the standard distribution. It's somewhat slower and a bit clumbsy to use, but you do get to control the memory it uses. How much it will save you will depend on your application.

One tip. You have to specify size (in bytes) for the keys, the values and the total table size. This often isn't a problem for the keys but can make it awkward and less memory efficient than it might be if most of your values are short but you need to pre-allocate extra length to accomodate the occasional long value. You can work around this limitation by storing references to scalars instead of the scalar itself. This means you only need to allocate 4-byte per value, but that the real value can be any length. Of course it adds a dereference to the equation. The other benefit of this is that by using references, you don't have to pad the value before assigning.

You will have to pad the keys to the pre-specified length though. Quite why this isn't done for you is not clear to me. However, it was a simple change to make to my copy of the module. I also made it so that over-long keys are automatically truncated. Slightly dangerous, but provided you ensure that the pre-allocated size is sufficient for your keys not to clash, it greatly simplifies using the module.

Caveat implementor should you decide to make this change yourself.

If the module works for you, it should be an easy adaption to your present code.


Examine what is said, not who speaks.

The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

Replies are listed 'Best First'.
Re: Re: A memory efficient hash, trading off speed - does it already exist?
by MarkM (Curate) on Feb 03, 2003 at 23:25 UTC

    BrowserUk suggests that Tie::SubstrHash be used as a potential solution...

    I recommend that this solution not be considered. tie()'d objects have overhead. In the cheapest case, we would be replacing a small basic hash with less than 4 entries with a tied hash, and an attached blessed string reference. No gain would be realized, and performance would suffer.

      tie()'d objects have overhead

      True, but the OP said that performance was not a concern.

      ... would be replacing a small basic hash with less than 4 entries with a tied hash, and an attached blessed string reference. No gain would be realized ...

      My thought was to replace the 120_000 element hash with a Tie::SubstrHash which does make a considerable saving in space. However, as you can't use a Tie::SubstrHash to store references, that throws the idea in the bin.

      I have an incomplete and broken version of the standard module that attempts to work around this particular restriction, but in the absence of the abilty to build XS/Inline-C, I am unable to complete that. There's no guarentee I could bring the idea to fruition anyway.


      Examine what is said, not who speaks.

      The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

        I think you were looking for Devel::Pointer. You'd have to find a way to manage the referent's reference count (B allows you to fiddle with it directly). If I were going to extend your module to use references I'd go this route - store the reference using the normal means and revivify it on demand.


        Seeking Green geeks in Minnesota

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://232405]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2024-03-29 06:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found