Crackers2 has asked for the wisdom of the Perl Monks concerning the following question:

I ran into some surprising (to me anyway) results with foreach and coderefs and list of constants. While investigating that I found a few more things that surprised me.

Let me start with the simplest case:

foreach my $x (0) { print $x++; }

Which unsurprisingly gives a Modification of a read-only value attempted message.

Changing this just a little bit to have (0,1) instead of just (0) doesn't make a difference.

Changing it to (0..1) does work though, which was my first surprise. Apparently lists like this created internally by perl aren't read-only. Still, this probably isn't news to most monks.

Where things got really interesting for me was when adding coderefs into the mix. Again starting with a simple case, there's no surprises:

my $cl1 = sub { foreach my $x (0) { print $x++; } }; $cl1->(); $cl1->();

This, as expected, gives the read-only error.

Going to the equivalent of the second case...

my $cl2 = sub { foreach my $x (0..1) { print $x++; } }; $cl2->(); $cl2->();

does not show any new surprises. The output is

0101

Changing this just one bit more...

my $cl3 = sub { foreach my $x (0..1,5..6) { print $x++; } }; $cl3->(); $cl3->();

is where things get strange. Instead of my expected
01560156
output, I got
01561267

In other words, it appears that the 0..1,5..6 list was generated once when the sub was first called (or defined even?), and then reused

I'm not sure why this only happens if you combine several ranges.

Personally I'd consider the fact that lists generated from ranges aren't read-only a bug. Without this the other cases would just go away.

Does anyone know why this is happening?

Update: Removed "closures" from the title since as ikegami has shown they're irrelevant

Update2: Condense output a bit for easier viewing

Update3:

On ikegami's optimization remark, I want to add one more testcase:

for (1..2) { foreach my $x (0..1,5..6) { print $x++; } foreach my $x (0..1,5..6) { print $x++; } }

which shows that it does keep a different copy for both loops.

Update4: Retitled per Ieronim

Replies are listed 'Best First'.
Re: Unexpected behaviour with constant lists
by ikegami (Patriarch) on Jul 20, 2006 at 18:33 UTC

    Closures are not relevant here:

    #for (1..2) { # foreach my $x (0) { print $x++; } # Error! #} for (1..2) { foreach my $x (0..1) { print $x++; } # 01, 01 } for (1..2) { foreach my $x (0..1,5..6) { print $x++; } # 0156, 1267 }

    I'm not sure why this only happens if you combine several ranges.

    Actually, it also applies with just a single range:

    for (1..2) { print map { $_++ } 0..1; # 01, 12 } for (1..2) { print map { $_++ } 0..1,5..6; # 0156, 1267 }

    Keep in mind that foreach (expr1..expr2) is optimized into a counting loop, so $x is not an alias to a constant list in the case of foreach my $x (0..1).


    You've apparently found an optimization. If you turn 0..1,5..6 into a non-constant expression, the expression rebuilds the list every time:

    for (1..2) { foreach my $x (map $_, 0..1,5..6) { print $x++; } # 0156, 0156 }

    Another fix is:

    for (1..2) { foreach (0..1,5..6) { my $x = $_; print $x++; } # 0156, 0156 }

    Personally I'd consider the fact that lists generated from ranges aren't read-only a bug.

    Aye. Lists deemed constant for optimization purposes should have its values flagged read-only.

    Update: Added a bit to make my explanations easier to follow. Adjusted my code to match the changes the OP did in his "Update 2".

      I'm not exactly sure what you mean by counting loop. Are you saying that

      foreach $x (expr1..expr2) { blah }

      is implemented more like

      for ($y=expr1 ; $y<expr2 ; $y++) { $x=$y; blah }

      internally ?

        Indeed. It's probably implemented identically to for (my $x=expr1 ; $x<=expr2 ; $x++) { blah }

        Update: Changed "<" to "<=".

Re: Unexpected behaviour with constant lists and foreach (and closures?)
by davido (Cardinal) on Jul 20, 2006 at 18:42 UTC

    This is an interesting edge case. I modified your code to eliminate $x, using localized $_ to see if that changed the behavior. It doesn't. Behold:

    use strict; use warnings; my $cl3 = sub { local $_; foreach (0..1,5..6) { print $_++ , "\n"; } }; $cl3->(); $cl3->(); $cl3->();

    This returns "015612672378". I ran it through B::Deparse to see what I wasn't seeing already. The deparsed version is like this:

    use warnings; use strict 'refs'; my $cl3 = sub { local $_; foreach $_ ((0, 1), (5, 6)) { print $_++, "\n"; } } ; &$cl3(); &$cl3(); &$cl3();

    If you run that, you get "Modification of a read-only value attempted at mytest.pl line 13." So deparse breaks your edge case by pre-enumerating the '..' lists.

    Update: ikegami is probably right; you've found an optimization, and a way to demonstrate it changing a script's expected behavior. And given the fact that it's probably an optimization you've bumped into, it isn't surprising that the deparsed version isn't able to "benefit" from the optimization, since it enumerates out the .. list.

    I'm curious: Every now and then someone posts one of these somewhat amazing but carefully constructed scenarios that exhibits an unexpected behavior or points out a bug. My curiosity is this, did you discover this while tracking down a real-world bug, while playing around with edge cases on your own, by reading the Perl source code, or was someone discussing it in some other forum (eg IRC or P5P) such that you caught wind of it and spilled it over into PerlMonks? I'm not making a value judgement, I always wonder how these things are first discovered. There have been a couple of oddities that I've uncovered while playing around with and refining obfuscations, for example.


    Dave

      I ran into this one when I was doing some benchmarking. Specifically, I had something like this:

      cmpthese(10000, { 'orig' => sub { foreach my $x ( 0..9, 'a'..'v', 'A'..'V' ) { $x = M +yMod::MapChar($x); }}, 'new' => sub { foreach my $x ( 0..9, 'a'..'v', 'A'..'V' ) { $x = M +yMod::MapChar2($x); }}, } );

      I first wrote it without the $x =, but was thinking that this might trigger some void optimization, so I added that part (without thinking about the aliasing implications obviously). This made those subs bomb out with "invalid parameter" errors when they received "10".

Re: Unexpected behaviour with constant lists and foreach (and closures?)
by Ieronim (Friar) on Jul 20, 2006 at 18:44 UTC
    I think that it means that foreach (#blah alah blah) created an anonymous array, that can be reused :) I don't know if it is a bug or a feature of Perl :)

    The anonymous array is not created only in the case of for(number..number), as ikegami mentioned, as such loops are replaced by counting ones.

    The shortest illustrating example:

    for (0..2) { foreach my $x (0..1, ()) { print +($x .= " wassap? ") . "\n"; } }

    It prints:
    0 wassap? 1 wassap? 0 wassap? wassap? 1 wassap? wassap? 0 wassap? wassap? wassap? 1 wassap? wassap? wassap?
    UPDATE:

    B::Deparse displays that it ocuurs only to arrays generated by the .. operator.

    UPDATE 2:

    I just realised that i used the feature of modifyable '..' arrays in my code :)

    print map { s/$/\n/;$_ } (0..1);
    works, while
    print map { s/$/\n/;$_ } (0,1);
    produces a fatal error :) So we can create one more example:
    for (0..1) { print map { s/$/!\n/;$_ } (0..1); }

    The bug is that anonymous arrays created by .. are globally scoped :) and I don't think it'a an optimisation for Perl.

    Finally, i think that the title of the node is incorrect :)). The good title would be Unexpected behavior of '..' lists. Or smth like this :)


         s;;Just-me-not-h-Ni-m-P-Ni-lm-I-ar-O-Ni;;tr?IerONim-?HAcker ?d;print