Re: Memory Error Printing Multiple Hashes Simultaneously

And here is a speedy reply to the perlbug by Gisle Aas:

This is not a memory leak.  The reason perl allocates memory during 
printing is that it converts all the 0 values in the hashes to strings
+ 
and stores these.  This upgrades your SvIV structs containing 0 into 
the much larger SvPV structs containing both 0 and "0" (see [1]). 

--Gisle 

[1] http://search.cpan.org/src/GAAS/illguts-0.09/index.html#sviv
[download]

I won't claim to fully understand, but I guess it means that Perl is changing the hash from storing simply integers to storing both integers & strings during printing. And this change occurs on the hash itself, increasing memory usage. Printing without string interpolation (or join prevents this "upgrading" and thus prevents all the memory usage. I guess it makes sense, but I'm just really surprised no one had come across this before!

Comment on Re: Memory Error Printing Multiple Hashes Simultaneously Select or Download Code

Replies are listed 'Best First'.
Re^2: Memory Error Printing Multiple Hashes Simultaneously by Anonymous Monk on Jan 31, 2006 at 23:36 UTC
One way to avoid this upgrade to strings is to use printf and ask for the numeric values of the hash values: `printf("%s\t%s\t%d\t%d\n", $key1, $key2, $hash1{$key1}{$key2}, $hash2{$key1}{$key2});` [download]	[reply] [d/l]
Re^2: Memory Error Printing Multiple Hashes Simultaneously by BrowserUk (Patriarch) on Feb 01, 2006 at 09:35 UTC
Glad you got an speedy explanation. It seems so obvious once it is pointed out, but then no one else here spotted the cause either:( It very non-intuative that the rates of memory growth you were seeing could be attributed to the simple act of stringifying previously numeric values. Once you know what to look for, it is easy to see the how quickly the memory usage grows when ths happens. A (minimum) of 70% growth for integers: `c:\Perl\test>p1 [0] Perl> use Devel::Size qw[ total_size ];; [0] Perl> @a = (0) x 1e6;; [0] Perl> print total_size( \@a );; 20000056 [0] Perl> $_.='' for @a;; [0] Perl> print total_size( \@a );; 34000056` [download] And a whopping 300% for floats. `c:\Perl\test>p1 [0] Perl> use Devel::Size qw[ total_size ];; [0] Perl> @a = ( 1.0 ) x 1e6;; [0] Perl> print total_size( \@a );; 24000056 [0] Perl> $_.='' for @a;; [0] Perl> print total_size( \@a );; 75000056` [download] Definitely one to remember if you are running close to the memory limits of your machine. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]
Re^3: Memory Error Printing Multiple Hashes Simultaneously by bernanke01 (Beadle) on Feb 01, 2006 at 18:14 UTC
Yup, and the actual use-case was floats (simplified down to integers for debugging). Thanks for all your help with this BrowserUk.	[reply]
Re^2: Memory Error Printing Multiple Hashes Simultaneously by demerphq (Chancellor) on Feb 01, 2006 at 15:28 UTC
Just to explain in more detail: `N:\>perl -MDevel::Peek -e"my @x=(1..2); Dump($x[0]); $y=join '-',@x; D +ump($x[0]); SV = IV(0x182b3e4) at 0x2257e8 REFCNT = 1 FLAGS = (IOK,pIOK) IV = 1 SV = PVIV(0x225ef4) at 0x2257e8 REFCNT = 1 FLAGS = (IOK,POK,pIOK,pPOK) IV = 1 PV = 0x22d3fc "1"\0 CUR = 1 LEN = 2` [download] The first shows `$x[0]` when its an SvIV. The second output is the same var after it has been stringified (which means its been "upgraded" to an SvPVIV). Note that there are now additional slots in the SV that are populated, containing such things as the stringified version of the integer,(PV) how long the string is, and how much of the allocated string is actually used. (LEN and CUR). What this means is that if you want to avoid the problem simply use the form `print join map { "$_" },@list;` [download] which will make the SV that gets upgraded be a temporary, leaving the original unchanged. My guess why it hasnt come up before is that probably its unusual for such a large structure to contain only SvIV's. So under normal circumstance id guess that if an OOM error occured it occured while building the structure itself and not when it was printed out. --- $world=~s/war/peace/g	[reply] [d/l] [select]
Re^3: Memory Error Printing Multiple Hashes Simultaneously by bernanke01 (Beadle) on Feb 01, 2006 at 18:12 UTC
Ahh, cool. Devel::Peak is much less scary when it's applied to a problem you already understand a bit. Thank you!	[reply]
Re^4: Memory Error Printing Multiple Hashes Simultaneously by demerphq (Chancellor) on Feb 01, 2006 at 19:10 UTC
Actually no, thank you for bringing this to our attention. This is a subtle issue that for those of us who need to build large structures and print them out would be well worth considering. I know I have scripts where I can apply this information when I next visit them. Anyway. Cheers. :-) --- $world=~s/war/peace/g	[reply]