Looping through a hash reference is creating a key...?

the_slycer has asked for the wisdom of the Perl Monks concerning the following question:

I could swear I've seen this behavior before, but can't find the post.

#!/usr/bin/perl
use warnings;
use strict;
use Data::Dumper;

my $questions = {
    1 => { foo => 'bar' },
    2 => { foo => 'bar' },
    3 => { foo => 'bar' }
};

print "(before):\n",Dumper($questions),"\n";

for (1 .. 4) {
    my $foo = $questions->{$_}{foo} || "daklfjsdalk";
}

print "(after): \n",Dumper($questions), "\n";
[download]

This returns something like:

(before):
$VAR1 = {
          1 => {
                 'foo' => 'bar'
               },
          2 => {
                 'foo' => 'bar'
               },
          3 => {
                 'foo' => 'bar'
               }
        };

(after): 
$VAR1 = {
          1 => {
                 'foo' => 'bar'
               },
          2 => {
                 'foo' => 'bar'
               },
          3 => {
                 'foo' => 'bar'
               },
          4 => {}
        };

And I'm not sure why, my question is twofold:
1) Is there a good reason why perl creates the reference (4)?
2) Is there a good reason why perl doesn't WARN that I am trying to access a value that doesn't exist? Note that even taking out the || "djkafdkljfa" does not produce a warning

The problem is appearing in a bit of code I am developing, and I can get around it, but this tripped me up while I was working on it (using exists($hashref->{key} to determine execution path).

ps:
Damn, I miss this place, I think it's time to start hanging out here again (especially now that I "do" perl for a living :))

Comment on Looping through a hash reference is creating a key...? Download Code

Replies are listed 'Best First'.
Re: Looping through a hash reference is creating a key...? by Roy Johnson (Monsignor) on Jun 10, 2004 at 18:12 UTC
Autovivification. Because you have to dereference `$questions{4}` to get to `$questions{4}{foo}`, the reference `$questions{4}` must be created. Update: Note that if the not-already-existing reference is not dereferenced, but is just read, no autovivification goes on, so your code would work as you'd like if you did: `for (1 .. 4) { my $foo = $questions->{$_} && $questions->{$_}{foo} \|\| "daklfjsdal +k"; }` [download] The PerlMonk `tr///` Advocate	[reply] [d/l] [select]
Re: Looping through a hash reference is creating a key...? by perrin (Chancellor) on Jun 10, 2004 at 18:19 UTC
Autovivification is the reason. If you want to avoid this, either use exists() before reading values or look at locked hashes using Hash::Util.	[reply]
Re^2: Looping through a hash reference is creating a key...? by Roy Johnson (Monsignor) on Jun 10, 2004 at 19:13 UTC
I'm going to quibble with your advice regarding `exists`, which could be right or wrong depending on how one reads it. (I don't doubt that your understanding is correct, but I think others might read it differently than you intended.) Autovivification (the creation of intermediate references) is not prevented by using `exists`, so `exists $h{1}{2}` still creates `$h{1}`. To prevent autovivification, one would have to test `exists $h{1}` and only if it were true, examine `$h{1}{2}`. However, they could just test whether `$h{1}` were true and get pretty much the same result -- if it didn't exist before, it won't exist after. The only difference in using `exists` is if `$h{1}` has a non-reference value like zero, empty string, or undef. It seems a little odd to dereference it in those cases. Perhaps a better idea would be to test it with `ref`. The PerlMonk `tr///` Advocate	[reply] [d/l] [select]
Re^3: Looping through a hash reference is creating a key...? by perrin (Chancellor) on Jun 10, 2004 at 19:19 UTC
Good point. I remember finding that out the hard way when I wrote some code that did exists() checks on deeply nested nodes in a huge data structure. My exists checks made the structure grow about 10MB in memory! Adding uglier but more careful code (`if (exists $h{1} && exists $h{1}->{2} && exists ...`) made the problem go away.	[reply] [d/l]
Re^4: Looping through a hash reference is creating a key...? by thor (Priest) on Jun 10, 2004 at 20:20 UTC
Re^5: Looping through a hash reference is creating a key...? by dave_the_m (Monsignor) on Jun 10, 2004 at 22:36 UTC
Some notes below your chosen depth have not been shown here
Re^2: Looping through a hash reference is creating a key...? by drewbie (Chaplain) on Jun 11, 2004 at 03:41 UTC
FYI, the hash locking feature only started in 5.8.0. From the docs. Restricted hashes 5.8.0 introduces the ability to restrict a hash to a certain set of keys. No keys outside of this set can be added. It also introduces the ability to lock an individual key so it cannot be deleted and the value cannot be changed. This is intended to largely replace the deprecated pseudo-hashes. And hooray for removing pseudohashes! I have the notes from Damian Conway's Advanced OO Perl talk (Boston 2001) that talks a bit about pseudohashes. They were a cool idea, but the implementation was very flawed. The more you read about them, the less interested you became.	[reply]
Re: Looping through a hash reference is creating a key...? by imcsk8 (Pilgrim) on Jun 11, 2004 at 04:16 UTC
i know that this is not an answer regarding autovivification, but i think this comment establishes a point. i think the problem is the approach you are choosing, if you are using numbers as hash keys, you should use an array (or an anonymous array reference) to store the hashes: (untested) `my @questions = ( { foo => 'bar' }, { foo => 'bar' }, { foo => 'bar' } );` [download] and get the values using a more natural loop: `foreach $hashref (@questions){ my $foo = $hashref->{foo} \|\| "daklfjsdalk"; }` [download] or `for(my $i = 0;$i<=$#questions;$i++){ my $foo = $questions[$i]->{foo} \|\| "daklfjsdalk"; }` [download] i know that this does not solve the issue, but sometimes different approaches can eliminate ambiguities on the code and the data results i hope this would be useful ignorance, the plague is everywhere --guttermouth	[reply] [d/l] [select]
Re: Looping through a hash reference is creating a key...? by Anonymous Monk on Jun 11, 2004 at 02:30 UTC
Perl does something called autovivication. How you get around it is exactly what you've done here using the 'exists' test. Autovivication can be your friend at times, but it can really bite you too.	[reply]
Re: Looping through a hash reference is creating a key...? by sacked (Hermit) on Jun 11, 2004 at 14:28 UTC
Also see Uri Guttman's article on autovivification. It includes some sample code to check for hash keys at any level of a multi-level data structure without triggering autovivification. (paraphrased from description in article) http://tlc.perlarchive.com/articles/perl/ug0002.shtml --sacked	[reply]
Re: Looping through a hash reference is creating a key...? by the_slycer (Chaplain) on Jun 11, 2004 at 19:29 UTC
Thank you all for the responses, just to address a few points brought up.. First and formost, the code above is not representative of the data structure I was using other than in a vague, hash of hashrefs type of way. I do use numbers, but theyre not sorted etc, still need to be in a hash :) Secondly, with regards to "exists", the problem became apparent because I was attempting to use exists as part of the flow of the code.. briefly: `$asked = exists($href->{$qid}); if ($asked) { #do stuff } else { #do other stuff } # later %data_structure = ( key => value, key2 => value2, key3 => $href->{$qid}{answer} \|\| 'stuff', #more assignments ); #do things with the data structure` [download] The first loop through the code would work fine, further loops would later would raise warnings (when doing data validation). Once I saw/replicated the problem, it was the lack of warnings on the first loop that raised my eyebrows. I hadn't heard of locked hashes, time to do some reading (though they don't look applicable for this particular issue at first glance)	[reply] [d/l]