in reply to Re^2: out of memory problem
in thread out of memory problem

You're right. The reuse of the address could be coincidence, but as the LEN stay 12, we can conclude that the buffer was never freed.

I've done a little bit of testing too:

sub foo{
    my $var = 'xyz' x 1_000_000;
}

sub bar{
    my $var2 = 'xyz' x 1_000_000;
}

warn "ONE\n";
foo();
warn "TWO\n";
foo();
warn "THREE\n";
bar();
warn "FOUR\n";

[download]

Running this through strace gives this output:

[...]
write(2, "ONE\n", 4)                    = 4
mmap2(NULL, 3002368, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
+-1, 0) = 0xb7956000
mmap2(NULL, 3002368, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
+-1, 0) = 0xb7679000
write(2, "TWO\n", 4)                    = 4
write(2, "THREE\n", 6)                  = 6
mmap2(NULL, 3002368, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
+-1, 0) = 0xb739c000
mmap2(NULL, 3002368, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 
+-1, 0) = 0xb70bf000
write(2, "FOUR\n", 5)                   = 5

[download]

This shows that perl does not allocate memory between "TWO" and "THREE" (the variable in sub foo gets reused), but it does allocate new memory before running bar, even though it could reuse $var's memory for $var2.

Including system('ps', '-Orss', '-p', $$) and die "$! $?"; in various places leads to the same conclusion without strace.

One question remains: When does perl free memory? I can't quite believe that every lexical variable that has ever been used results in large stale memory blocks. This would be against the spirit of this excerpt from perlsyn:

You wouldn't want memory being free until you were done using it, or kept around once you were done. Automatic garbage collection takes care of this for you.

Replies are listed 'Best First'.
Re^4: out of memory problem
by ikegami (Patriarch) on Sep 13, 2008 at 15:56 UTC

    but it does allocate new memory before running bar, even though it could reuse $var's memory for $var2.

    No, that would defy the optimization of not deallocating it in the first place.

    For Perl to know when $var is not currently being used, it would have to 1) place $var on a free list when it stops being used and 2) remove it from the free list when $var is in use again. Those two operations are known as deallocation and allocation, exactly what the optimization is trying to avoid!

    Keep in mind that each variable consists of two memory blocks, three if they have a string buffer associated with them. And that's just for non-magical scalars. A lot of CPU time is being saved.

    One question remains: When does perl free memory?

    Anon SVs are freed. SVs that become anon (such as return values) are freed. SVs passed to undef have their attached buffer freed.

    Keep in mind that all of this is the result of an (undocumented?) optimization. It shouldn't be relied upon. It's just that you can't rely on it not happening.

[reply]
[d/l]
[select]

      Suppose I write a module which provides a "frobnicate" function:

      sub frobnicate {
    my ($string) = @_;
    $string =~ s/foo/bar/g;
    return $string;
}

      [download]

      Suppose I write a program that runs for a really long time, and uses "frobnicate" only once during its initialization, and it passes a really long string to it.

      Should I expect that $string will hold this long string for the program's lifetime? Am I supposed to undef every single variable to avoid this? (Which would pretty much nullify this "optimization", but it seems to be the only way to avoid the memory leak.)

      Update: Replaced the code of the example function.

[reply]
[d/l]
[select]
        Yes, as far as I know
[reply]

        The way to avoid that problem when you know you are dealing with large scalars, (arrays and hashes too), is to use pass by reference. Either: 

        sub frobnicate {
    $_[0] =~ s/foo/bar/g;
    return;
}

        [download]

        Or: 

        sub frobnicate {
    my ($ref) = @_;
    $$ref =~ s/foo/bar/g;
    return;
}

        [download]

        Either way you avoid a lot of copying and the lingering redundant memory usage.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom