Re^6: Tracking down an Lvalue bug?

Related to ref to read-only alias ... why??

I don't think so.

This demonstrates the problem (bug):

perl -E"$x=chr(0); $x x= 100e6; <>; $r = \substr $x, 50e6; <>"
[download]

Watch the process memory with ProcessManager:

When it pauses the first time memory usage is ~98MB. Hit enter
Now the memory usage shoots up to ~145MB.
All that's changes is that it has taken a substr ref to the second half of the string.

I suspect it is a result of this "fix".

But I believe that fix is wrong, and that the original bug is spurious. (At least in part.)

If you do this:

my $ref;
{
    my $string = "123";
    $ref = \$string;
}

## Here $string persists.
[download]

No one is surprised that (the memory allocated to) $string persists beyond the block.

Why should this be any different:

my $ref;
{
    my $string = "123";
    $ref = \substr $string, 1, 1;
}

## This was the (IMO false) "bug" scenario.
[download]

In this case -- when the string goes out of scope -- I could make arguments for 3 possible outcomes:

The memory allocated to $string persists:
$ref continues to refer (directly) to the memory allocate to $string.
Those parts of $string outside of $ref are inaccessible and irrecoverable until $ref is GC's.
The memory allocated to $string is GC'd; $ref becomes undef.
Probably difficult to orchestrate.
The memory allocated to $string persists until $ref is used the next time:
At which point the extraneous (pre and/or post fix) bits of $string are GC'd leaving $ref pointing at just that which it references.
This would require retaining a pointer back to 'parent' string with the lvalue ref, and when the lvalue magic fires it checks the refcount of its parent.
If that refcount is 1 -- meaning it holds the only reference to it, it frees off the extraneous parts of the parent.

Of the three, I'd prefer C, but would find A completely in keeping with Perl's philosophy.

Duplicating the referenced memory of every lvalue ref and the having to copy it back each time it is modified -- just to cover off a really obscure scenario that is at worst, only mildly anomalous -- is crazy.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Comment on Re^6: Tracking down an Lvalue bug? Select or Download Code

Replies are listed 'Best First'.
Re^7: Tracking down an Lvalue bug? (!fix) by tye (Sage) on Mar 20, 2012 at 14:05 UTC
For what its worth, I concur. - tye	[reply]
Re^7: Tracking down an Lvalue bug? by ikegami (Patriarch) on Mar 21, 2012 at 01:30 UTC
The fix has no effect on: `my $ref; { my $string = "123"; $ref = \substr $string, 1, 1; } ## This was the (IMO false) "bug" scenario.` [download] The fix is for: `{ my $string = "123"; my $ref = \substr $string, 1, 1; }` [download] All the patch does (or is suppose to do) is avoid saving the SV inside of the OP, something that causes `$string` until the end of the program in the second snippet. Do you have a way of automating the test so I can use git bisect to find when the behaviour changed (if it did).	[reply] [d/l] [select]
Re^8: Tracking down an Lvalue bug? by BrowserUk (Patriarch) on Mar 21, 2012 at 01:56 UTC
Do you have a way of automating the test Is a windows solution suitable? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^9: Tracking down an Lvalue bug? by ikegami (Patriarch) on Mar 21, 2012 at 02:01 UTC
No, not only have I've never built Perl on Windows, I doubt Porting/bisect.pl will work there. In fact, even using modules is problematic. I guess I should start by manually checking the releases around my lvalue patch.	[reply]
Re^10: Tracking down an Lvalue bug? by BrowserUk (Patriarch) on Mar 21, 2012 at 02:32 UTC
Re^7: Tracking down an Lvalue bug? by ikegami (Patriarch) on Mar 21, 2012 at 03:35 UTC
The patches for RT#67838 are is found in: `5.13.4+ 5.14.0+` [download] I ran your test program with ActivePerl 5.14.0 (32 bit) on Windows, and noticed NO memory increase. (Steady at 99MB.) In fact, I saw in increases only up to that version of Perl. `perl -E"$x=x; $x x= 100e6; <>; $r = \substr $x, 50e6; <>; ord $$r; <>" AP 5.12.4 32-bit 99M 148M 148M AP 5.14.0 32-bit 99M 99M 148M AP 5.14.2 32-bit 99M 99M 148M` [download] There will be a memory increase if you read the string `$$r`, but that's how magic works in Perl. I looked at the source code at the time of the patch. There is no intentional "prefetching", and testing shows no accidental prefetching: `>\progs\perl5142-ap1402\bin\perl -MDevel::Peek -E"$x=chr(0); $x x= 100 +; Dump substr $x, 50;" SV = PVLV(0x4de51c) at 0x497b04 REFCNT = 1 FLAGS = (TEMP,GMG,SMG) IV = 0 NV = 0 PV = 0 <----- No buffer MAGIC = 0x4cc694 MG_VIRTUAL = &PL_vtbl_substr MG_TYPE = PERL_MAGIC_substr(x) TYPE = x TARGOFF = 50 TARGLEN = 50 TARG = 0x4a932c SV = PV(0x25603c) at 0x4a932c REFCNT = 2 FLAGS = (POK,pPOK) PV = 0x4b36cc "\0\0\0...\0\0\0"\0 CUR = 100 LEN = 104` [download] In contrast with a version that does grow as soon as substr is called: `>\progs\perl5124-ap1205\bin\perl -MDevel::Peek -E"$x=chr(0); $x x= 100 +; Dump substr $x, 50;" SV = PVLV(0x319e4c) at 0x3bf54 REFCNT = 1 FLAGS = (PADMY,GMG,SMG,pPOK) IV = 0 NV = 0 PV = 0x2f42dc "\0\0\0...\0\0\0"\0 <----- CUR = 50 LEN = 52 MAGIC = 0x327f94 MG_VIRTUAL = &PL_vtbl_substr MG_TYPE = PERL_MAGIC_substr(x) TYPE = x TARGOFF = 50 TARGLEN = 50 TARG = 0x2f875c SV = PV(0x36034) at 0x2f875c REFCNT = 2 FLAGS = (POK,pPOK) PV = 0x330f6c "\0\0\0...\0\0\0"\0 CUR = 100 LEN = 104` [download] $ref continues to refer (directly) to the memory allocate to $string. Neither `$ref` nor `$$ref` ever refer directly to `$string`'s PV. They can't because the address of `$string`'s buffer can change as `$string` changes. This would require retaining a pointer back to 'parent' string with the lvalue ref `$$ref` does have a reference to `$string`. Probably difficult to orchestrate. Actually, that's easy. Just weaken `$$ref`'s reference to `$string`. At which point the extraneous (pre and/or post fix) bits of $string are GC'd leaving $ref pointing at just that which it references. One can't free the start of a memory block. I don't think one can even free the end of a memory block. The string would have to be copied to a new buffer in order to shrink the buffer. Doable, but it would require a temporary "doubling" of memory. It would also be uncharacteristic of Perl. Perl intentially avoids freeing memory left and right. Shrinking a buffer would be a first!	[reply] [d/l] [select]
Re^8: Tracking down an Lvalue bug? by BrowserUk (Patriarch) on Mar 23, 2012 at 01:14 UTC
ActivePerl 5.14.0 (32 bit) on Windows Confirming that the behaviour is also changed on 64-bit 5.14 for windows. There will be a memory increase if you read the string $$r, but that's how magic works in Perl. I accept that is how it currently works, but question whether is should or has to work that way. If the reference is only ever used for reading, duplicating the reference memory is a complete waste of time (cpu & memory). At the very least it should be possible to defer the duplication until a mutating operation is performed upon it. Even then, I question whether it is necessary to duplicate the substring, and then copy the modified version back over the original. If a temporary SV* (be it a TARG or LVTARG or just a mortal SV* created for the purpose) was initialised so that its PV pointed directly at the original strings PV buffer, and it used the OOK trick, CUR & LEN to make that temporary SV* point to just the referenced substring in-place. And then that temp SV was passed to the mutating operator, everything should work as it would with a normal SV. When the mutating operator completes, the PUT magic gets control and can fix up the original (referenced) Sv to reflect any changes. But, I am aware that this is the kind of thing we could argue back forth, round & round and down a dozen blind alleys and never reach a conclusion. The only way to avoid that is to prove my theory. And that is going to take time. I'm also currently working on a machine that has a dodgy motherboard and an increasingly frequent habit of blue screening at the slightest provocation. I have a new motherboard and ram on its way, but till then, I'm not getting into anything deep because of the risk of loosing code though random failures. So bear with me. I'll either come back with proof of concept code or an acknowledgement that I couldn't make it work. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]