Re^2: memory use when merging tied hashrefs

you can iterate over the hashes using each

Nice! Memory doesn't budge when using each, but the program's slightly too slow, so I'll trade memory for speed. Good to know for a cloudy day. I did away with hash3 and merged hash2 into hash1:

while (my ($k,$v) = each %$hash2) { $hash1->{$k} = $v }

Thanks for the lesson

Comment on Re^2: memory use when merging tied hashrefs

Replies are listed 'Best First'.
Re^3: memory use when merging tied hashrefs by Eily (Monsignor) on Nov 13, 2019 at 17:18 UTC
Another possible middleground solution is to use a for loop on the keys. `$hash3->{$_} = $hash1->{$_} for keys %$hash1;` This potentially means that only the keys will be stored in a temporary list (not sure about the exact impact though, even more so when tied hashes are involded). Also, to improve the speed, you can force perl to preallocate a hash with `keys %hash = 200;`, that's a little bit tricky to use because the right value is the number of buckets in the hash, not the number of keys. See the end of Scalar values about that syntax. To choose the correct number, you can use Hash::Util::bucket_ratio() after doing the job once and getting an idea of the expected size of the hash (do that on the final hash, not a tied one though!). Without that step, perl may first allocate a hash that is too small, and need to copy it into a bigger one when you add too much data. I doubt this happens often with "only" 8000 keys, but it's always worth a try. (Thanks to choroba for helping find the relevant documentation :D)	[reply] [d/l] [select]
Re^3: memory use when merging tied hashrefs by Eily (Monsignor) on Nov 13, 2019 at 15:56 UTC
If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db?	[reply]
Re^4: memory use when merging tied hashrefs by Anonymous Monk on Nov 13, 2019 at 20:22 UTC
If $hash1 is tied to a database, does that mean you are adding the records from $hash2 into the first db? Yes. That was dumb. Next customer! :-)	[reply]