Re^2: Memory Error Printing Multiple Hashes Simultaneously

Thanks BrowserUK, I thought I was going crazy. I wonder if the extra factor might be the tabs themselves, as I can still get an out-of-memory error by replacing the join with manual tabs:

use strict;

my %hash1;
my %hash2;
my %hash3;

my @data = (0 .. 5000);

for (my $i = 0; $i < scalar(@data); $i++ ) {
        for (my $j = $i; $j < scalar(@data); $j++) {
                $hash1{ $data[$i] }{ $data[$j] } = 0;
                $hash2{ $data[$i] }{ $data[$j] } = 0;
                $hash3{ $data[$i] }{ $data[$j] } = 0;
                }
        }

print "Finished building hashes\n";

while ( my( $key1, $val1 ) = each %hash1 ) {
    while ( my( $key2, $val2 ) = each %$val1 ) {
        die 'Autovivify' unless exists $hash1{ $key1 }
                            and exists $hash1{ $key1 }{ $key2 };
        print "$key1\t$key2\t$hash1{ $key1 }{ $key2 }\t$hash2{ $key1 }
+{ $key2 }\t$hash3{ $key1 }{ $key2 }\n";
    }
}
[download]

Comment on Re^2: Memory Error Printing Multiple Hashes Simultaneously Download Code

Replies are listed 'Best First'.
Re^3: Memory Error Printing Multiple Hashes Simultaneously by BrowserUk (Patriarch) on Jan 31, 2006 at 17:37 UTC
Well, I'm almost ashamed to tell you that I tried this :), but it doesn't matter what character you use in the catenation, whether with join or interpolation into a string, it still causes the memory growth. I even tried calling a function to do the output, passing the values as a list and joining them inside the function, and still it occurs? I do have a work around for you. `print map{ $_, "\t" } $key1, $key2, $hash1{ $key1 }{ $key2 }, $hash2{ $key1 }{ $key2 }, $hash3{ $key1 }{ $key2 }; print "\n";` [download] Replace your print line with the above, and the memory growth will completely disappear. It isn't an exact replacement for the join, as you will get an extra tab at the end of the line. If that is a problem then you can go for manually interspersing the tabs: `print $key1, "\t", $key2 "\t", $hash1{ $key1 }{ $key2 }, "\t", $hash2{ $key1 }{ $key2 }, "\t", $hash3{ $key1 }{ $key2 }, "\n";` [download] That both these avoid the memory growth indicates that the problem comes from building the single output string, which is something I guess, but this is bread and butter Perl code and it would surely have been noticed between 5.6.2 and now if it was that simple. I have completely failed to reproduce the problem outside of two nested while loops accessing compound hashes using variables as keys. Every simplification I apply to reduce your code to a testcase and the problem disappears. Anyway, I hope the above insights will allow you to get on with solving your problem; but do please raise a perlbug and let the guys that understand this stuff have a go at a proper solution/explanation. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: Memory Error Printing Multiple Hashes Simultaneously
by BrowserUk (Patriarch) on Jan 31, 2006 at 17:37 UTC

Well, I'm almost ashamed to tell you that I tried this :), but it doesn't matter what character you use in the catenation, whether with join or interpolation into a string, it still causes the memory growth. I even tried calling a function to do the output, passing the values as a list and joining them inside the function, and still it occurs?

I do have a work around for you.

 print map{ $_, "\t" } $key1, $key2,
    $hash1{ $key1 }{ $key2 }, 
    $hash2{ $key1 }{ $key2 },
    $hash3{ $key1 }{ $key2 };
 print "\n";
[download]

Replace your print line with the above, and the memory growth will completely disappear. It isn't an exact replacement for the join, as you will get an extra tab at the end of the line. If that is a problem then you can go for manually interspersing the tabs:

 print $key1, "\t", $key2 "\t", 
    $hash1{ $key1 }{ $key2 }, "\t",
    $hash2{ $key1 }{ $key2 }, "\t",
    $hash3{ $key1 }{ $key2 }, "\n";
[download]

That both these avoid the memory growth indicates that the problem comes from building the single output string, which is something I guess, but this is bread and butter Perl code and it would surely have been noticed between 5.6.2 and now if it was that simple.

I have completely failed to reproduce the problem outside of two nested while loops accessing compound hashes using variables as keys. Every simplification I apply to reduce your code to a testcase and the problem disappears.

Anyway, I hope the above insights will allow you to get on with solving your problem; but do please raise a perlbug and let the guys that understand this stuff have a go at a proper solution/explanation.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]
[d/l]
[select]