in reply to Re^2: string to more compact format
in thread string to more compact format

Packing the keys will make barely any difference to the storage requirements of your hash as shown below. In the first case, a hash containing 2e6 x 50-byte keys takes 70MB. In the second case, 2e6 x 12-byte keys 67MB: 

$hash{ sprintf "%050d", $_ } = 1 for 0 .. 2e6;;
print total_size \%hash;;
70183684

$hash{ sprintf "%012d", $_ } = 1 for 0 .. 2e6;;
print total_size \%hash;;
67660852

[download]

If all you want to do is store the strings so you can iterate over them, storing the uncompressed strings in an array will save far more space (30MB versus 67/70MB): 

$a[ $_ ] = sprintf "%050d", $_ for 0 .. 2e6;;
print total_size \@a;;
30408448

[download]

You also save the cost of compressing and decompressing.

There are also other ways of storing and iterating your 2e6 strings that take even less memory but are just as easy and efficient to iterate.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP an inspiration; A true Folk's Guy

Replies are listed 'Best First'.
Re^4: string to more compact format
by Boetsie (Sexton) on May 17, 2010 at 14:29 UTC
    Hmm strange that storing a 50 byte key hash is just a fraction larger than a 12-byte key hash. Why is that? I can't store my strings into an array. Since sometimes strings occur more than once and i should keep track of that.
[reply]
      Hmm strange that storing a 50 byte key hash is just a fraction larger than a 12-byte key hash. Why is that?

      Because for each key/value pair, there are 40 bytes (32-bit, more on 64-bit) of overhead in addition to the key and value data. See Hash structure illustration. So for short keys, most of the space used by a hash is in the internal construction, not the keys & values themselves.

      I can't store my strings into an array. Since sometimes strings occur more than once and i should keep track of that.

      Then use the hash, but don't bother with the compression because you won't gain anything from it. 70MB isn't such a lot these days.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

      I can't store my strings into an array. Since sometimes strings occur more than once...

      I'm curious, if the strings can occur "more than once" then how do you use the hash? Would you keep track of the number of occurences in the value of the hash?

      ack Albuquerque, NM
[reply]

In Section Seekers of Perl Wisdom