tadman has asked for the wisdom of the Perl Monks concerning the following question:
It's amazing how something so minor can turn Perl from a happy, simple programming language into an evil cancerous growth that consumes all available memory on your system. Not even strict or -w could save you.
It all started with some code that used used to extract a particular value from a simple HoH:
my $foo = { foo => { bar => 'baz' } };
foreach (keys %foo)
{
if (defined($foo->{$_}->{foo})
&& defined($foo->{$_}->{foo}->{bar}))
{
return $foo->{$_}->{foo}->{bar};
}
return;
}
To prevent auto-vivification, it tests carefully and returns the appropriate value. This worked well in practice, but in order to make the function more useful, it now had
to search an AoHoH and return the first match. A simple change, of course. How about this:
my $foo = [
{ foo => { bar => 'baz' } },
];
foreach my $foo_entry (@$foo)
{
foreach (keys %$foo_entry)
{
if (defined($foo->{$_}->{foo})
&& defined($foo->{$_}->{foo}->{bar}))
{
return $foo->{$_}->{foo}->{bar};
}
}
return;
}
Can you spot the error? strict can't. When I run this, I'm into swap space in a matter of seconds, and...then...things...get...very...slow.
Somehow I messed up the variable names, since in the actual source they were slightly more similar and there was more clutter which prevented me from seeing this right away.
If the code is "fixed", it runs fine.
What is strange is that this is all caused by using a AoHoH instead of a AoH. An example using just AoH causes a warning:
my $foo = [
{ foo => 'bar' },
];
foreach my $foo_entry (@$foo)
{
foreach (keys %$foo_entry)
{
if (defined($foo->{$_}->{foo})
&& defined($foo->{$_}->{foo}->{bar}))
{
return $foo->{$_}->{foo}->{bar};
}
}
return;
}
Something is falling through the cracks here, no?
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion
by strat (Canon) on May 14, 2002 at 15:59 UTC
|
if( exists($foo->{$_} and exists($foo->{$_}->{bar} ) {
...
}
Best regards,
perl -le "s==*F=e=>y~\*martinF~stronat~=>s~[^\w]~~g=>chop,print" | [reply] [d/l] |
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion
by Matts (Deacon) on May 14, 2002 at 20:23 UTC
|
Yep, it's a "bug" in the psuedo hashes code. Remember how psuedo hashes work - they're an array ref where the first entry is a hashref identifying the entries. Unfortunately perl checks every arrayref for access like this...
Anyway, what's happening the value it gets from the first entry is a ref, but perl tries to treat it as a number. As you know, a ref looks like: HASH(0x804b55c). Now when we numify that, we get 134_526_303. So perl tries to access the entry somewhere past the 134 millionth entry. And we know that perl tries to autovivify that entry by extending the array that far.
I think this has been fixed in bleedperl, but I can't find the reference to the bug now.
Hope that helps. | [reply] |
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion
by ferrency (Deacon) on May 14, 2002 at 18:11 UTC
|
I think in your second example, your if statement is testing the wrong variable. Compare with this code:
my $foo = [
{ foo => { bar => 'baz' } },
];
foreach my $foo_entry (@$foo)
{
foreach (keys %$foo_entry)
{
if (defined($foo_entry->{$_}->{foo})
&& defined($foo_entry->{$_}->{foo}->{bar}))
{
return $foo_entry->{$_}->{foo}->{bar}
+;
}
}
return;
}
In the inner loop, $_ is a key of %$foo_entry. I'm not sure why your code passes strict, since $foo should contain an array ref and not a hash ref; I'd expect the if statement to bomb with "not a hashref".
I'm also confused about your question: Do you know the answer, and you're testing us, or have you still not figured out what's wrong?
Thanks :)
Update: On FreeBSD, perl 5.6.0, I get:
% perl hoh.pl
Out of memory during "large" request for 1073745920 bytes at hoh.pl line 15.
So, Yes it tries to gobble up lots of memory :) But luckily it fails all at once instead of eating swap and bombing the entire system.
Alan | [reply] [d/l] [select] |
|
I mentioned that I had "fixed" it, which involved doing exactly what you pointed out. It was kind of odd that strict didn't get this one, so there were no bombs, apart from the feverish memory consumption.
I didn't post the fix because I wanted to see if it was obvious or not, which considering it took the third reply to "solve", I suppose it really isn't.
As a point of curiosity, does the broken example chew up all your RAM? I'm using Perl 5.6.1 on RedHat 7.2 i386 and it's swap city.
| [reply] |
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion
by alien_life_form (Pilgrim) on May 14, 2002 at 16:38 UTC
|
$aa=[ {qq=>{rr=>1}} ];
DB<16> p defined($aa->{pippo})
No such pseudo-hash field "pippo" at (eval 19)
DB<17> p defined($aa->{qq})
DB<18> p defined($aa->{pippo}->{lillo})
No such pseudo-hash field "pippo" at (eval 21)
DB<19> p defined($aa->{qq}->{lillo})
WTF? Pseudohash uh? (which should be deprecated anyway...)
Really weird.
Cheers,
alf
You can't have everything: where would you put it? | [reply] [d/l] |
Re: Automagic Array to Hash Conversion? Pseudo-Hash Explosion
by Util (Priest) on May 14, 2002 at 18:45 UTC
|
I agree with Alf; it is a bug in Perl's pseudohash support.
You happen to have an arrayref of which the first element is
a hashref, and that is the definition of a pseudohash.
This triggers the same all-you-can-eat memory bug:
my $foo;
$foo->[0]->{foo}->{bar} = 'baz';
$foo->{foo}->{foo}->{bar} = 'baz';
As ferrency points out, your examples are confusing. I
suspect that something was lost when you trimmed them for
posting. Even the first example, which (I think) you meant
to be correct, fails on the data you posted. Either
$foo needs an additional level of hashref
(becoming HoHoH), or the foreach loop is not
needed. Also, your return statements should be
one level further out in all three examples.
I was working on an example of a less error-prone
way to handle your AoHoH, when I realized that something was
amiss in the example code. Can you clarify? | [reply] [d/l] [select] |
|
| [reply] [d/l] [select] |
|
|