dda has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

Why the following code does not process elements added inside the loop:
#!/usr/bin/perl use strict; use warnings; my %h = ( 1 => 1111111, 2 => 2222222, 3 => 3333333, 4 => 4444444, 5 => 5555555, ); foreach (sort keys %h) { print "Processing $_\n"; my $new = $_*10; $h{$new} = $new; print " Addding $new\n"; next; } exit;
Probably, the list for looping is populated before the loop starts? How will it work with large amounts of data?

--dda

Replies are listed 'Best First'.
Re: foreach loop question
by dws (Chancellor) on Apr 16, 2003 at 18:41 UTC
    Why the following code does not process elements added inside the loop ...

    Consider that

    foreach ( sort keys %h ) { ... $h{$new} = $new;
    is equivalent to
    @sorted = sort keys %h; foreach ( @sorted ) { ... $h{$new} = $new;
    The keys have been extracted from the hash before being sorted and processed. Any keys you add after that aren't "visible" to the foreach.

    By the way, the "next;" is redundant.

      Well, thanks. But without "sort" it works the same way. I wonder how it processes large arrays - always creates a copy?

      --dda

        Well, thanks. But without "sort" it works the same way.

        I should have been more explicit. It's "keys" that creates the new array, not the sort.

        If you want to process all of the keys and values in a hash without creating a new array to hold the hash keys, look into "each".

        while ( ($key, $value) = each %h ) { ...
        This doesn't create an additional array. Note, though, that it's unsafe to manipulate the hash while you're iterating over it. The proviso in the docs reads:
        If you add or delete elements of a hash while you’re iterating over it, you may get entries skipped or duplicated, so don’t. Exception: It is always safe to delete the item most recently returned by "each()".
Re: foreach loop question
by dmitri (Priest) on Apr 16, 2003 at 18:44 UTC
    > Probably, the list for looping is populated before the loop starts?
    
    Yes.
    

    You might want to try

    while (my ($key, $value) = each(%h)) { }
    But I am not sure if you'll get all your new keys.

    P.S. next; at the end of foreach block is redundant.

      You might want to try {code with each} But I am not sure if you'll get all your new keys.

      Actually, the docs for each specifically say that you should not do this - as each returns hash elements in an essentially random order (or at least in an unpredictable order), so doing something like this is playing with fire :).

      CU
      Robartes-

        I always figured that each and related functions keys and values iterated over the hash from start to finish with the order being determined by the hashing algorithm. Can an internals person shed some light on what determines this "essentially random order"?


        "The dead do not recognize context" -- Kai, Lexx
        That's what I meant by "I am not sure if you'll get all your new keys." I guess it did not come across that way.
      About "next": I added it while experimenting with the code. Thanks :)

      --dda

Re: foreach loop question
by robartes (Priest) on Apr 16, 2003 at 18:45 UTC
    You are correct - a foreach loop builds a temporary list, and iterates over that. That means that when you loop over the keys of a hash using foreach, entries added inside the loop are not iterated over.

    For 'one by one' iteration over a hash, have a look at each, but you cannot use that to add elements to the hash on the fly while iterating over it either (well, you can, but it's your foot :) ).

    You'll have to iterate over the hash keys again if you want to get at the new values, I think.

    CU
    Robartes-

      "You are correct - a foreach loop builds a temporary list, and iterates over that." I don't think this is true. Try this:
      my @list = (1,2,3,4,5); foreach (@list) { print $_."\n"; push @list,$_.$_; }
      This makes an infinite loop (at least until your memory runs out), so it appears that foreach is actually using the original list

        This is documented in perlsyn.

        If any part of LIST is an array, foreach will get very confused if you add or remove elements within the loop body, for example with splice. So don't do that.

Re: foreach loop question
by jonadab (Parson) on Apr 16, 2003 at 20:01 UTC

    Setting aside the way Perl handles this specific case, ask yourself how a computer program could keep track of which items in a collection it had already processed, if items can be inserted into the middle of the collection during processing. The only way I know is to build a second list. There are two ways to do this: you can keep a list of ones you've already done, or a list of ones you haven't done yet. If your keys (in this case, the hash keys) have a defined order, you can in theory limit your list of not-done-yet ones somewhat by having "all the ones with a key greater than this value n" be one (possibly implicit) always-final entry in the list, adding entries to the list only when that doesn't cover them, and raising the value of n whenever it's the only item in the list. This requires some work on your part, though, and in most cases is probably not worth it. If your footprint is a significant issue, though, it is an option. Something along the lines of this...

    $n = minimum_key(\%h); # minimum_key left as an exercise while (@n or ($n<maximum_key(\%h))) { # also maximum_key if not (@n) { push @n, keys_between(\%h, $n, $n+1); # Yep, keys_between is also left as an exercise. $n++; } my $thiselement = shift @n; process_element($thiselement); }

    If you do something like this, be sure to be consistent with your inequalities (e.g., if an element is equal to n, it either has to be in @n or covered by $n but not both). Also of course if you insert anything into the hash as part of the processing you have to test whether the key is less than $n and if so add it to @n.


    for(unpack("C*",'GGGG?GGGG?O__\?WccW?{GCw?Wcc{?Wcc~?Wcc{?~cc' .'W?')){$j=$_-63;++$a;for$p(0..7){$h[$p][$a]=$j%2;$j/=2}}for$ p(0..7){for$a(1..45){$_=($h[$p-1][$a])?'#':' ';print}print$/}
Re: foreach loop question
by Limbic~Region (Chancellor) on Apr 16, 2003 at 22:05 UTC
    dda,
    I have been monitoring this thread for most of the day and as far as I can tell, no one has stated the obvious.

    You would have (or could create) an infinite loop.

    Cheers - L~R

    Update: As fletcher_the_dog pointed this behavior does create the infinite loop for an array (not hash), and as pointed out by Mr. Muskrat that this is a known deficiency and it should be avoided.

      I have been monitoring this thread for most of the day and as far as I can tell, no one has stated the obvious.

      There is such a thing as letting people discover their next problem. If we solve it for them before they encounter it, how will they learn?

Re: foreach loop question
by mod_alex (Beadle) on Apr 17, 2003 at 09:00 UTC
    #! c:/perl/bin/perl.exe use strict; use warnings; my %h = ( 1 => 1111111, 2 => 2222222, 3 => 3333333, 4 => 4444444, 5 => 5555555, ); my $ind = 0; my $item = (sort(keys(%h)))[$ind++]; do { print "Processing $item \n"; my $new = $item*10; $h{$new} = $new; print " Addding $new\n"; $item = (sort(keys(%h)))[$ind++]; } while (1); print 'good luck \n';
Re: foreach loop question
by mod_alex (Beadle) on Apr 17, 2003 at 08:57 UTC
    #! c:/perl/bin/perl.exe use strict; use warnings; my %h = ( 1 => 1111111, 2 => 2222222, 3 => 3333333, 4 => 4444444, 5 => 5555555, ); my $ind = 0; my $item = (sort(keys(%h)))[$ind++]; do { print "Processing $item \n"; my $new = $item*10; $h{$new} = $new; print " Addding $new\n"; $item = (sort(keys(%h)))[$ind++]; } while (1); print 'good luck \n';

    20030417 Edit by Corion: Added code tags

Re: foreach loop question
by aquarium (Curate) on Apr 17, 2003 at 12:38 UTC
    changing array/hash size whilst looping it will give you strange results. it seems that the discussion is centering on how weird can you get...but if you actually want a solution to your problem: implement your own (get module) for a linked list. a linked list is meant to be manipulated that way. Even if you do hack a solution with the hash that finally works for you, it's bound to break on a different platform/version of perl. Chris