Automagic Array to Hash Conversion? Pseudo-Hash Explosion

tadman has asked for the wisdom of the Perl Monks concerning the following question:

It's amazing how something so minor can turn Perl from a happy, simple programming language into an evil cancerous growth that consumes all available memory on your system. Not even strict or -w could save you.

It all started with some code that used used to extract a particular value from a simple HoH:

my $foo = { foo => { bar => 'baz' } };

foreach (keys %foo)
{
        if (defined($foo->{$_}->{foo})
         && defined($foo->{$_}->{foo}->{bar}))
        {
                return $foo->{$_}->{foo}->{bar};
        }

        return;
}
[download]

To prevent auto-vivification, it tests carefully and returns the appropriate value. This worked well in practice, but in order to make the function more useful, it now had to search an AoHoH and return the first match. A simple change, of course. How about this:

my $foo = [
        { foo => { bar => 'baz' } },
];

foreach my $foo_entry (@$foo)
{
        foreach (keys %$foo_entry)
        {
                if (defined($foo->{$_}->{foo})
                 && defined($foo->{$_}->{foo}->{bar}))
                {
                        return $foo->{$_}->{foo}->{bar};
                }
        }

        return;
}
[download]

Can you spot the error? strict can't. When I run this, I'm into swap space in a matter of seconds, and...then...things...get...very...slow.

Somehow I messed up the variable names, since in the actual source they were slightly more similar and there was more clutter which prevented me from seeing this right away.

If the code is "fixed", it runs fine.

What is strange is that this is all caused by using a AoHoH instead of a AoH. An example using just AoH causes a warning:

my $foo = [
        { foo => 'bar' },
];

foreach my $foo_entry (@$foo)
{
        foreach (keys %$foo_entry)
        {
                if (defined($foo->{$_}->{foo})
                 && defined($foo->{$_}->{foo}->{bar}))
                {
                        return $foo->{$_}->{foo}->{bar};
                }
        }

        return;
}
[download]

Something is falling through the cracks here, no?

Comment on Automagic Array to Hash Conversion? Pseudo-Hash Explosion Select or Download Code

Replies are listed 'Best First'.
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by strat (Canon) on May 14, 2002 at 15:59 UTC
Your problem might be auto-vivification. With defined, you just check if a value of a hash is defined, not if it is existing. (And the value of a hash may be existing but undef without problems). Better use something like: `if( exists($foo->{$_} and exists($foo->{$_}->{bar} ) { ... }` [download] Best regards, perl -le "s==F=e=>y~\martinF~stronat~=>s~[^\w]~~g=>chop,print"	[reply] [d/l]
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by Matts (Deacon) on May 14, 2002 at 20:23 UTC
Yep, it's a "bug" in the psuedo hashes code. Remember how psuedo hashes work - they're an array ref where the first entry is a hashref identifying the entries. Unfortunately perl checks every arrayref for access like this... Anyway, what's happening the value it gets from the first entry is a ref, but perl tries to treat it as a number. As you know, a ref looks like: HASH(0x804b55c). Now when we numify that, we get 134_526_303. So perl tries to access the entry somewhere past the 134 millionth entry. And we know that perl tries to autovivify that entry by extending the array that far. I think this has been fixed in bleedperl, but I can't find the reference to the bug now. Hope that helps.	[reply]
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by ferrency (Deacon) on May 14, 2002 at 18:11 UTC
I think in your second example, your if statement is testing the wrong variable. Compare with this code: `my $foo = [ { foo => { bar => 'baz' } }, ]; foreach my $foo_entry (@$foo) { foreach (keys %$foo_entry) { if (defined($foo_entry->{$_}->{foo}) && defined($foo_entry->{$_}->{foo}->{bar})) { return $foo_entry->{$_}->{foo}->{bar} +; } } return; }` [download] In the inner loop, `$_` is a key of `%$foo_entry`. I'm not sure why your code passes strict, since $foo should contain an array ref and not a hash ref; I'd expect the `if` statement to bomb with "not a hashref". I'm also confused about your question: Do you know the answer, and you're testing us, or have you still not figured out what's wrong? Thanks :) Update: On FreeBSD, perl 5.6.0, I get: % perl hoh.pl Out of memory during "large" request for 1073745920 bytes at hoh.pl line 15. So, Yes it tries to gobble up lots of memory :) But luckily it fails all at once instead of eating swap and bombing the entire system. Alan	[reply] [d/l] [select]
Re^2: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by tadman (Prior) on May 14, 2002 at 18:33 UTC
I mentioned that I had "fixed" it, which involved doing exactly what you pointed out. It was kind of odd that strict didn't get this one, so there were no bombs, apart from the feverish memory consumption. I didn't post the fix because I wanted to see if it was obvious or not, which considering it took the third reply to "solve", I suppose it really isn't. As a point of curiosity, does the broken example chew up all your RAM? I'm using Perl 5.6.1 on RedHat 7.2 i386 and it's swap city.	[reply]
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by alien_life_form (Pilgrim) on May 14, 2002 at 16:38 UTC
Greetings, mmmmmm..... `$aa=[ {qq=>{rr=>1}} ]; DB<16> p defined($aa->{pippo}) No such pseudo-hash field "pippo" at (eval 19) DB<17> p defined($aa->{qq}) DB<18> p defined($aa->{pippo}->{lillo}) No such pseudo-hash field "pippo" at (eval 21) DB<19> p defined($aa->{qq}->{lillo})` [download] WTF? Pseudohash uh? (which should be deprecated anyway...) Really weird. Cheers, alf You can't have everything: where would you put it?	[reply] [d/l]
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by Util (Priest) on May 14, 2002 at 18:45 UTC
I agree with Alf; it is a bug in Perl's pseudohash support. You happen to have an arrayref of which the first element is a hashref, and that is the definition of a pseudohash. This triggers the same all-you-can-eat memory bug: `my $foo; $foo->[0]->{foo}->{bar} = 'baz'; $foo->{foo}->{foo}->{bar} = 'baz';` [download] As ferrency points out, your examples are confusing. I suspect that something was lost when you trimmed them for posting. Even the first example, which (I think) you meant to be correct, fails on the data you posted. Either `$foo` needs an additional level of hashref (becoming HoHoH), or the `foreach` loop is not needed. Also, your `return` statements should be one level further out in all three examples. I was working on an example of a less error-prone way to handle your AoHoH, when I realized that something was amiss in the example code. Can you clarify?	[reply] [d/l] [select]
Re^2: Automagic Array to Hash Conversion? Pseudo-Hash Explosion by tadman (Prior) on May 14, 2002 at 20:04 UTC
The second example is the one that crashes Perl for me. It looks a little strange, I'll admit, because it's supposed to be inside a function, and `$foo` is supposed to be passed in via `@_`. Here's an idea of how it looked: `sub bar { my ($foo) = @_; foreach my $foo_entry (@$foo) { foreach (keys %$foo_entry) { if (defined($foo->{$_}->{foo}) && defined($foo->{$_}->{foo}->{bar})) { return $foo->{$_}->{foo}->{bar}; } } } return; }` [download] During simplification, I accidentally put the `return` inside the foreach, but it doesn't matter. The `defined` call kills it right away. This code is busted, as I forgot to change the internal references to `$foo_entry`. What surprised me was that a) `strict` didn't care, and b) it caused Perl to go crazy. After some experimentation, I came up with that example as the least amount of code required to reproduce the problem. The data given is complete junk, the real data much more complex, but it too is the bare minimum required to produce the problem. The real structure does have additional levels of "oH". Pseudo-hashes are dangerous to your health then?	[reply] [d/l] [select]