Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

According to Advanced Perl Programming, I see that hash keys always employ the string representation.

I have a hash with integer keys -- lots of (sparse) integer keys. I am running into a memory limitation. Could I save memory by storing the MIME::Base64 version of the integer key? (Or perhaps using pack to create a 4-byte or 8-byte string.)

Thanks!

Replies are listed 'Best First'.
Re: are perl hash keys always strings?
by Fletch (Bishop) on Dec 22, 2010 at 16:56 UTC

    If you've got that big a hash then the savings you're going to get from diddling the keys probably aren't going to help. Look into BerkeleyDB or the like and move your hash onto disk instead.

    The cake is a lie.
    The cake is a lie.
    The cake is a lie.

Re: are perl hash keys always strings?
by ikegami (Patriarch) on Dec 22, 2010 at 16:55 UTC
    For true hashes, yes. Magical hashes can extend this.

    Could I save memory by storing the MIME::Base64 version of the integer key? (Or perhaps using pack to create a 4-byte or 8-byte string.)

    $ perl -MMIME::Base64 -E'say length for $ARGV[0], encode_base64($ARGV[0]), pack("N", $ARGV[0]), pack("Q", $ARGV[0]) ' 1234 4 9 4 8
Re: are perl hash keys always strings?
by BrowserUk (Patriarch) on Dec 22, 2010 at 17:44 UTC

    If the string representation of your integers is longer than their packed representation, then you will save a few bytes per key, but the savings will be minimal. A few hundred kilobytes at most:

    undef $h{ $_ } for 1e6 .. 2e6;; print total_size( \%h );; 34617643 undef $i{ pack 'V', $_ } for 1e6 .. 2e6;; print total_size( \%i );; 34231634 undef $j{ $_ } for 100e6 .. 101e6;; print total_size( \%j );; 34883144 undef $k{ pack 'V', $_ } for 100e6 .. 101e6;; print total_size( \%k );; 34239854

    And the relative saving is less still if you are assigning values to the keys.

    If you described your application and the nature/distribution of your keys in more detail, there are often ways of storing such data that has significantly lower memory costs. Though these usually trade space for speed.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: are perl hash keys always strings?
by JavaFan (Canon) on Dec 22, 2010 at 17:21 UTC
    They are "strings", although they aren't internally stored as Perl strings. You can use any other type as key, but it will be stringified - and that's usually a one-way operation. (For instance, you typically cannot retrieve an reference after it has been stringified (you can if it's still around on the same memory address, and you throw the appropriate XS magic to it - but if it has been garbage collected, or threads have given it a new virtual address, you're out of luck)). In particular, keys will only return the stringified forms of the objects, not the original ones.