http://qs1969.pair.com?node_id=510411

blazar has asked for the wisdom of the Perl Monks concerning the following question:

Update: solved! Well, it will be solved...

I was playing around with tie and interpolation (for obfu purposes, FWIM), and I got what seems to me as a strange result:

#!/usr/bin/perl -l

use strict;
use warnings;

sub TIESCALAR { bless \my $i, shift }
sub FETCH { ${shift,}++ }

tie my $s, 'main';
print "$s$s$s";
print "$s$s$s";

tie my $t, 'main';
print " $t$t$t";
print " $t$t$t";

__END__

[download]

This gives me the following output: 

112
445
 012
 345

[download]
so it seems that something wrong, or at least unexpected, happens when a tied variable appears at the beginning of an interpolating string.

Remark: please note that this is not an artifact of the minimalistic nature of the example, i.e. of the fact that I use main as the class providing the implementation of the variable, and that I do not provide a STORE method. I tried with more complex stuff and indeed the behaviour shown above persists.

Also note that with arrays everything works as one would naively expect - consider: 

#!/usr/bin/perl -l

use strict;
use warnings;

sub TIEARRAY { bless \my $i, shift }
sub FETCHSIZE { 3 }
sub FETCH { ${shift,}++ }

tie my @s, 'main';
print "@s - @s - @s";

__END__

[download]
which gives me: 
0 1 2 - 3 4 5 - 6 7 8

[download]

Replies are listed 'Best First'.
Re: tie() weirdness
by Perl Mouse (Chaplain) on Nov 21, 2005 at 10:15 UTC
    Seems like a bug to me. At first I thought that there might have been a weird optimization going on (but if so, you'd expect '111' and '222' to be printed) - but if you trace the program using -Dt, you notice that FETCH is called three times.

    I'd perlbug it.

    Perl --((8:>*
[reply]

      Done!

      I received two answers pasted hereafter. According to the former it has been solved in bleædperl; according to the latter it doesn't really have to do with tie but with the use of "the same variable more than once in an expression, where use of that variable has side effect". This sheds some light on the extremely concise reply by Anonymous Monk, since the thread referenced there goes into some detail discussing related topics.

      To be fair the second email also claims that it "isn't actually a bug", which IMHO does conflict both with almost everyone here's perception (certainly it does not DWIM - now it may just be a misconceived dwimmery, but since it does apply as intended in practically identical situations, this doesn't seem to be the case!) and the claim of the first one that it has been solved.

      The answers
      Answer #1
       
      Date: Mon, 21 Nov 2005 09:47:02 -0800
From: Rafael Garcia-Suarez via RT <perlbug-followup@perl.org>
Subject: [perl #37722] interpolated tie()d variable weirdness 

This has been fixed in bleadperl by change #26185.

      [download]
      Answer #2
       
      Date: Mon, 21 Nov 2005 11:36:54 -0800
From: Dave Mitchell via RT <perlbug-followup@perl.org>
Subject: Re: [perl #37722] interpolated tie()d variable weirdness

On Mon, Nov 21, 2005 at 03:21:13AM -0800, Michele Dondi wrote:
> I noticed a strange and appearently inconsistent behaviour (to the b
+est of
> my knowledge non documented) when a tie()d variable is put at the *b
+eginning*
> of an iterpolating string. Minimal example code follows:
> 
> 
> #!/usr/bin/perl -l
> 
> use strict;
> use warnings;
> 
> sub TIESCALAR { bless \my $i, shift }
> sub FETCH { ${shift,}++ }
> 
> tie my $s, 'main';
> print "$s$s$s";
> print "$s$s$s";
> 
> tie my $t, 'main';
> print " $t$t$t";
> print " $t$t$t";
> 
> __END__
> 
> 
> Output:
> 
> 
> 112
> 445
>   012
>   345
> 
> 

Thanks for the report, but this isn't actually a bug!
Similar sorts of effects can be seen without tie, eg

    $ perl -le 'print ++$x . ++$x . ++$x'
    223
    $ perl -le 'print " " . ++$x . ++$x . ++$x'
     123
    $

Note that interpolated strings are just short hand for string concats,
+ and
in fact that's how perl handles them internally, ie
    " $s$s$s" is equivalent to (("".$s).$s).$s

and in general when you when you start using the same variable more th
+an
once in an expression, where use of that variable has side effect,
unexpected things will generally happen.

-- 
Never do today what you can put off till tomorrow.

      [download]
[reply]
[d/l]
[select]
Re: tie() weirdness
by robin (Chaplain) on Nov 21, 2005 at 15:27 UTC
    Yes, this is certainly a bug. Here is what's happening:

    Update: This diagnosis is mistaken — see below.

    • The way $foo++ is implemented is that the value of $foo is copied to a temporary SV (the op's targ), the original value is then incremented; and finally the top element of the stack is pointed at the targ.
    • In this example, the second time FETCH is called, the targ gets overwritten with the new value. But there's already a pointer to it on the stack.
    I guess one solution would be to use a temporary mortal instead of the targ.
[reply]
      I guess one solution would be to use a temporary mortal instead of the targ

      i just tried that. It doesn't work, because the mortals get freed on the return from the FETCH routine. I should have realised that would happen.

      Ouch! I wonder how it can be fixed then?

      Update: for completeness I should point out that, although it is indeed true that mortals get freed on return from the FETCH routine, that isn't the (only) reason this doesn't fix the problem. A correct (I hope!) diagnosis and patch can be found in the node below.

[reply]
      Update: This diagnosis is mistaken — see below.
      I read all of your posts in this thread (replying just to this one out of convenience). Since you submitted a fix yourself, you may be interested to read the replies I got after having submitted a bug report.
[reply]
        Thanks for the pointer.

        Update: In case it isn't clear, Rafael was talking about my fix when he said that it's fixed in bleadperl.

        There is truth in what dave_the_m says, but the fact is that this behaviour is highly surprising, and can be fixed. It's better to make things behave sensibly than to spin elaborate explanations of why they don't. :-)

[reply]
Re: tie() weirdness
by robin (Chaplain) on Nov 21, 2005 at 16:46 UTC
    My diagnosis above isn't right. The real problem is with the ordering of operations in pp_concat: the left and right operands are both fetched before either is copied. I have a patch that fixes it: 
    --- pp_hot.c.orig       2005-11-21 16:22:04.000000000 +0000
+++ pp_hot.c    2005-11-21 17:00:57.000000000 +0000
@@ -148,11 +148,14 @@
     dPOPTOPssrl;
     bool lbyte;
     STRLEN rlen;
-    const char *rpv = SvPV_const(right, rlen); /* mg_get(right) happe
+ns here */
-    const bool rbyte = !DO_UTF8(right);
+    const char *rpv;
+    bool rbyte;
     bool rcopied = FALSE;
 
     if (TARG == right && right != left) {
+       /* mg_get(right) may happen here ... */
+       rpv = SvPV_const(right, rlen);
+       rbyte = !DO_UTF8(right);
        right = sv_2mortal(newSVpvn(rpv, rlen));
        rpv = SvPV_const(right, rlen);  /* no point setting UTF-8 here
+ */
        rcopied = TRUE;
@@ -179,6 +182,11 @@
            SvUTF8_off(TARG);
     }
 
+    /* or mg_get(right) may happen here */
+    if (!rcopied) {
+       rpv = SvPV_const(right, rlen);
+       rbyte = !DO_UTF8(right);
+    }
     if (lbyte != rbyte) {
        if (lbyte)
            sv_utf8_upgrade_nomg(TARG);
--- t/op/tie.t.orig     2005-11-21 16:50:25.000000000 +0000
+++ t/op/tie.t  2005-11-21 16:52:07.000000000 +0000
@@ -578,3 +578,10 @@
 print $h,"\n";
 EXPECT
 3.3
+########
+sub TIESCALAR { bless {} }
+sub FETCH { shift()->{i} ++ }
+tie $h, "main";
+print $h.$h;
+EXPECT
+01

    [download]
    Just need to do some testing, then I'll send it to p5p.

    Update: I've added tests and fixed a unicode bug that I accidentally reintroduced (as per the updated patch in this node), and submitted the patch.

    Second update: the patch was applied (as change #26185), then Rafael noticed a subtle bug and reverted it (as change #26190), then I fixed the bug and the fixed patch was applied as change #26192. Phew!

[reply]
[d/l]
Re: tie() weirdness
by ikegami (Patriarch) on Nov 21, 2005 at 14:31 UTC

    Concatenation of a list can be implemented as 

    $str = concat($s1, $s2);
$str = concat($str, $s3);
$str = concat($str, $s4);

    [download]

    Notice the first case is special. Maybe what you're seeing is related to this?

[reply]
[d/l]

      It may well be, since

      print $s . $s . $s;       # still produces the "error",
print '' . $s . $s . $s;  # makes it go away,

# But!

print join '', $s, $s, $s;   # is fine,

      [download]

      The latter should still be essentially a concatenation in which the first element of the list is the tied variable. So what we're observing should result from an interaction between the tie mechanism and the concatenation operator.

      BTW: it is well known that when a bare variable is passed as the first argument of a join, it is re-evaluated every time it's needed, and

      print join $s, ('') x 4;

      [download]

      gives me (with the code of Re^2: tie() weirdness)

      FETCH called!
FETCH called!
FETCH called!
FETCH called!
123

      [download]

      which is unexpected too, but can be understood in terms of an extra-evaluation.

[reply]
[d/l]
[select]
Re: tie() weirdness
by Anonymous Monk on Nov 21, 2005 at 11:11 UTC
[reply]
      Hmmm, I gave a peek into it and it doesn't seem to be directly related to tie, although some discussion about tied variables does pop out in the middle of thread; but the latter is quite long and something very relevant may be "hidden" somewhere in its depths. Otherwise it may just be me overlooking something obvious. Whatever, can you point us to a more specific, ehm, point of it?
[reply]
Re: tie() weirdness
by Anonymous Monk on Nov 21, 2005 at 11:17 UTC
[reply]

      But from that thread seems to refer to a (possible) bug in tied arrays, whereas I found a (possibly) anomalous behaviour with tied scalars.

      However, this particular post from that thread brings me to

      Both hint at the fact that tied vars may... well, evaluate twice under some circumstances. In particular the former seems to refer to a different situation, but the latter portraits exactly the same one as mine. But the same article claims that it was a 5.6.1 specific bug and that it should have been patched - here's the relevant quote:

      I searched p5p and it appears to have been reported and patched in development versions --- I just wanted to let people here know so they might avoid some confusion if they run into it. A relevant p5p link (from which the patch is linked) is: here

      May it be that the patch never made its way in the next releases or that the same bug has been reintroduced subsequently?

      Remark: more importantly, it seems that the FETCH method is not called twice: 

      #!/usr/bin/perl -l

use strict;
use warnings;

sub TIESCALAR { bless \my $i, shift }
sub FETCH { warn "FETCH called!\n"; ${shift,}++ }

tie my $s, 'main';
print "$s$s$s";

__END__

      [download]
      Output: 
      FETCH called!
FETCH called!
FETCH called!
112

      [download]

[reply]
[d/l]
[select]

        It certainly is a strange thing. This post lists some more strange things and no solution. Something certainly was changed (between v5.6 and v5.8)... what I don't know.

        If you run your latest code on v5.6.1 then you'll see that FETCH is called 4 times, not three times. (and it prints 123).

        If you change it into print $s . $s . $s; then you get 123 on v5.6, and 112 on v5.8.

        Change it into: print "" . $s . $s . $s; and you get 012. (on v5.6 and v5.8)

        Change it into: print "$s$s" . "$s$s" . "$s$s" and you get 124578 on v5.6 and 113355 on v5.8.

        There is another case in which the scalar is evaluated twice (and that I didn't immediatly see in here/the links posted here): hash keys on v5.6. If the key does not exist then it will re-evaluate the scalar. And the last value will be used for the key. This seems to be fixed in v5.8.

        If this was 'normal' code I would strongly advice against using such a construct, since there are too many cases where it can go wrong...

        And if you look at the perlapi then you will find function that evaluate it twice aswell... (for example SvIV and SvIVx)

        Update: Some other 'strange' cases:

        • print $s . "$s$s" : v5.6: FETCH is called 5 times and the output is 412. v5.8: FETCH called 3 times and the output: 211.
        • print $s . "$s" . $s: v5.6: output 203. v5.8 output: 102. (The output of print $s.$s.$s is mentioned above)

[reply]
[d/l]
[select]
Re: tie() weirdness
by Anonymous Monk on Nov 21, 2005 at 10:53 UTC
    so it seems that something wrong, or at least unexpected, happens when a tied variable appears at the beginning of an interpolating string
    Where from is your expectation?
[reply]

Back to Seekers of Perl Wisdom