Re: memory use when merging tied hashrefs

It might depend on which version of perl you are using, but one way to avoid using a temporary list (%$hash1, %$hash2 is flattened to a list of pairs before being added to the anonymous hash) you can iterate over the hashes using each.

while (my ($key, $value) = each %$hash1)
{
  $hash3{$key} = $value;
}
[download]

Same for $hash2. This also depends on how the tied hash is implemented but it's worth a try...

Edit: or you could also tie hash3 to fetch data either from hash1 or hash2 ... really depends on what you are trying to achieve and what are your contraints.

Comment on Re: memory use when merging tied hashrefs Download Code

Replies are listed 'Best First'.
Re^2: memory use when merging tied hashrefs by Anonymous Monk on Nov 13, 2019 at 14:50 UTC
you can iterate over the hashes using each Nice! Memory doesn't budge when using each, but the program's slightly too slow, so I'll trade memory for speed. Good to know for a cloudy day. I did away with hash3 and merged hash2 into hash1: while (my ($k,$v) = each %$hash2) { $hash1->{$k} = $v } Thanks for the lesson	[reply]
Re^3: memory use when merging tied hashrefs by Eily (Monsignor) on Nov 13, 2019 at 17:18 UTC
Another possible middleground solution is to use a for loop on the keys. `$hash3->{$_} = $hash1->{$_} for keys %$hash1;` This potentially means that only the keys will be stored in a temporary list (not sure about the exact impact though, even more so when tied hashes are involded). Also, to improve the speed, you can force perl to preallocate a hash with `keys %hash = 200;`, that's a little bit tricky to use because the right value is the number of buckets in the hash, not the number of keys. See the end of Scalar values about that syntax. To choose the correct number, you can use Hash::Util::bucket_ratio() after doing the job once and getting an idea of the expected size of the hash (do that on the final hash, not a tied one though!). Without that step, perl may first allocate a hash that is too small, and need to copy it into a bigger one when you add too much data. I doubt this happens often with "only" 8000 keys, but it's always worth a try. (Thanks to choroba for helping find the relevant documentation :D)	[reply] [d/l] [select]
Re^3: memory use when merging tied hashrefs by Eily (Monsignor) on Nov 13, 2019 at 15:56 UTC
If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db?	[reply]
Re^4: memory use when merging tied hashrefs by Anonymous Monk on Nov 13, 2019 at 20:22 UTC
If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db? Yes. That was dumb. Next customer! :-)	[reply]