Hello,

This is a follow up to an earlier query, where I ran out of memory trying to output a very large hash. After somebody pointed me at autovivification as a potential culprit, I started narrowing down to a minimal test-case. I believe that this test-case rules out autovivification, but I'm not sure what it does suggest.

Here is a script that works correctly on my machine (AIX 5.2, Perl 5.8.7). Note that the code prints to screen, so call it as perl -w test.pl > test.txt.

use strict;

my %hash1;
my %hash2;

my @data = (0 .. 5000);

for (my $i = 0; $i < scalar(@data); $i++ ) {
        for (my $j = $i; $j < scalar(@data); $j++) {
                $hash1{ $data[$i] }{ $data[$j] } = 0;
                $hash2{ $data[$i] }{ $data[$j] } = 0;
                }
        }


while ( my ($key1, $val1) = each %hash1 ) {
        while ( my ($key2, $val2) = each %$val1 ) {
                print join(
                        "\t",
                        $key1,
                        $key2,
                        $hash1{$key1}{$key2},
                        $hash2{$key1}{$key2}
                        ), "\n";
                }
        }
[download]

Now here is a script that crashes with Out of memory! on my machine. The only different I can see between these two is that the first one has only two hashes and the second one three.

use strict;

my %hash1;
my %hash2;
my %hash3;

my @data = (0 .. 5000);

for (my $i = 0; $i < scalar(@data); $i++ ) {
        for (my $j = $i; $j < scalar(@data); $j++) {
                $hash1{ $data[$i] }{ $data[$j] } = 0;
                $hash2{ $data[$i] }{ $data[$j] } = 0;
                $hash3{ $data[$i] }{ $data[$j] } = 0;
                }
        }

print "Finished building hashes\n";

while ( my ($key1, $val1) = each %hash1 ) {
        while ( my ($key2, $val2) = each %$val1 ) {
                print join(
                        "\t",
                        $key1,
                        $key2,
                        $hash1{$key1}{$key2},
                        $hash2{$key1}{$key2},
                        $hash3{$key1}{$key2}
                        ), "\n";
                }
        }
[download]

Note that I've verified that the crash occurs *after* all three hashes are populated. Further, if I delete hash keys after I've processed them the script runs to completion:

use strict;

my %hash1;
my %hash2;
my %hash3;

my @data = (0 .. 5000);

for (my $i = 0; $i < scalar(@data); $i++ ) {
        for (my $j = $i; $j < scalar(@data); $j++) {
                $hash1{ $data[$i] }{ $data[$j] } = 0;
                $hash2{ $data[$i] }{ $data[$j] } = 0;
                $hash3{ $data[$i] }{ $data[$j] } = 0;
                }
        }

print "Finished building hashes\n";

while ( my ($key1, $val1) = each %hash1 ) {
        while ( my ($key2, $val2) = each %$val1 ) {
                print join(
                        "\t",
                        $key1,
                        $key2,
                        $hash1{$key1}{$key2},
                        $hash2{$key1}{$key2},
                        $hash3{$key1}{$key2}
                        ), "\n";
                }
        delete $hash1{$key1};
        delete $hash2{$key1};
        delete $hash3{$key1};
        }
[download]

So as far as I can tell, this shows that the error occurs when trying to output multiple large HoHs with identical keys simultaneously. I can't quite see why memory would be getting used in this process, though!!

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.