in reply to Memory Error Printing Multiple Hashes Simultaneously

I concur. There is sustained and heavy memory allocation going on in the second loop for which I can see no apparent reason. There is no autovivification of hash elements occuring. You have discovered another memory leak and the culprit is join(*).

If you replace your second loop with 

while ( my( $key1, $val1 ) = each %hash1 ) {
    while ( my( $key2, $val2 ) = each %$val1 ) {
        print( ##join( "\t",
            $key1,
            $key2,
            $hash1{ $key1 }{ $key2 },
            $hash2{ $key1 }{ $key2 },
            $hash3{ $key1 }{ $key2 }
        );
    }
}

[download]

The code runs to completion and shows zero memory growth during the second loop.

The problem also exists in 5.6.2. Raise a perlbug.

*Update: It's more complicated than just join. Seems it needs both join and a reference to a compound hash plus some unknown factor to cause the leak.

If you remove the join, or the hash reference, or make either of the keys a constant and the leak does not occur! 

while ( my( $key1, $val1 ) = each %hash1 ) {
    while ( my( $key2, $val2 ) = each %$val1 ) {
        die 'Autovivify' unless exists $hash1{ $key1 }
                            and exists $hash1{ $key1 }{ $key2 };
        print( join "\t", $key1, $key2, $hash1{ $key1 }{ $key2 }, );
    }
}

[download]
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Memory Error Printing Multiple Hashes Simultaneously
by bernanke01 (Beadle) on Jan 31, 2006 at 16:59 UTC

    Thanks BrowserUK, I thought I was going crazy. I wonder if the extra factor might be the tabs themselves, as I can still get an out-of-memory error by replacing the join with manual tabs:

    use strict;

my %hash1;
my %hash2;
my %hash3;

my @data = (0 .. 5000);

for (my $i = 0; $i < scalar(@data); $i++ ) {
        for (my $j = $i; $j < scalar(@data); $j++) {
                $hash1{ $data[$i] }{ $data[$j] } = 0;
                $hash2{ $data[$i] }{ $data[$j] } = 0;
                $hash3{ $data[$i] }{ $data[$j] } = 0;
                }
        }

print "Finished building hashes\n";

while ( my( $key1, $val1 ) = each %hash1 ) {
    while ( my( $key2, $val2 ) = each %$val1 ) {
        die 'Autovivify' unless exists $hash1{ $key1 }
                            and exists $hash1{ $key1 }{ $key2 };
        print "$key1\t$key2\t$hash1{ $key1 }{ $key2 }\t$hash2{ $key1 }
+{ $key2 }\t$hash3{ $key1 }{ $key2 }\n";
    }
}

    [download]
[reply]
[d/l]

      Well, I'm almost ashamed to tell you that I tried this :), but it doesn't matter what character you use in the catenation, whether with join or interpolation into a string, it still causes the memory growth. I even tried calling a function to do the output, passing the values as a list and joining them inside the function, and still it occurs?

      I do have a work around for you. 

       print map{ $_, "\t" } $key1, $key2,
    $hash1{ $key1 }{ $key2 }, 
    $hash2{ $key1 }{ $key2 },
    $hash3{ $key1 }{ $key2 };
 print "\n";

      [download]

      Replace your print line with the above, and the memory growth will completely disappear. It isn't an exact replacement for the join, as you will get an extra tab at the end of the line. If that is a problem then you can go for manually interspersing the tabs: 

       print $key1, "\t", $key2 "\t", 
    $hash1{ $key1 }{ $key2 }, "\t",
    $hash2{ $key1 }{ $key2 }, "\t",
    $hash3{ $key1 }{ $key2 }, "\n";

      [download]

      That both these avoid the memory growth indicates that the problem comes from building the single output string, which is something I guess, but this is bread and butter Perl code and it would surely have been noticed between 5.6.2 and now if it was that simple.

      I have completely failed to reproduce the problem outside of two nested while loops accessing compound hashes using variables as keys. Every simplification I apply to reduce your code to a testcase and the problem disappears.

      Anyway, I hope the above insights will allow you to get on with solving your problem; but do please raise a perlbug and let the guys that understand this stuff have a go at a proper solution/explanation.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom