Mini-Tutorial: Perl's Memory Management

This post answers a few common questions and misconceptions about Perl's memory management.

Lexical variables don't get freed at end of scope

Lexical variables don't get freed at end of scope. They are cleared on scope exit to give you the apperance of a fresh variable. This is an optimisation that should be transparent to you.

On scope exit, if anything prevents the variable from being resused (e.g. if there are remaining references to the variable), the existing variable becomes anonymous and a new variable is allocated.

Clearing a variable does not free the memory it uses

It was previously mentioned that variables on cleared on scope exit. Variables can also be cleared as follows:

$s = undef;
@a = ();
%h = ();
[download]

Clearing a variable frees no memory directly. The buffer of a string is marked as being unused, but it remains allocated for later use. The underlying array of pointers in arrays and hashes is cleared, but it remains allocated for later use. This is an optimisation that should be transparent to you.

As a result of a variable being cleared, referenced variables (incl array elements) have their refcount lowered, and that may free those variables. But not the variable being cleared.

`undef` does not free a variable

undef can be used to clear a variable.

undef $s;
undef @a;
undef %h;
[download]

undef does do more than clear the variable. It also frees any buffers the variable has.

$ perl -MDevel::Peek -e'
    $s = "abc"; Dump $s;
    $s = undef; Dump $s;
    undef $s;   Dump $s;
'
SV = PV(0x814fb00) at 0x814f69c
  REFCNT = 1
  FLAGS = (POK,pPOK)         <-- Var contains a string
  PV = 0x81651b8 "abc"\0
  CUR = 3
  LEN = 4
SV = PV(0x814fb00) at 0x814f69c
  REFCNT = 1
  FLAGS = ()                 <-- Var is undefined
  PV = 0x81651b8 "abc"\0     <-- String buffer still
  CUR = 3                         allocated after clearing
  LEN = 4
SV = PV(0x814fb00) at 0x814f69c
  REFCNT = 1
  FLAGS = ()                 <-- Var is undefined
  PV = 0                     <-- undef freed the buffer
[download]

This function is rarely useful.

However, undef will never free a variable. Clearing the variable may result in the reduction of the ref counts of other variables, which may free those variables. But not the variable that was passed to undef.

Freeing variables may result in no change in available system memory

You can't rely on memory being returned to the system, but it can happen.

If and when memory is returned to the OS is dependant on the memory allocation mechanism perl was compiled to use. You may also see differences between system using the same memory allocation mechanism. It's also possible that a mechanism will only ever release the large blocks it allocated.

The default memory allocation mechanism varies by OS. You are more likely to see memory being released to the OS on Windows.

Comment on Mini-Tutorial: Perl's Memory Management Select or Download Code

Replies are listed 'Best First'.
Re: Mini-Tutorial: Perl's Memory Management by wfsp (Abbot) on Oct 28, 2009 at 11:26 UTC
Lexical variables don't get freed at end of scope. They are cleared on scope exit... From then on in I was lost. What the difference is between "freed" and "cleared" or "end" and "exit" is in this context defeats me (and I'm not absolutely sure about lexical). Obscure in the extreme without a far deeper explanation of the subject. And that is just the second paragraph. :-) The title says "Mini-Tutorial: Perl's Memory Management". I would hope this is not included as or referred to as a faq/tutorial for scope or references, there are far friendly ways to do this.	[reply]
Re^2: Mini-Tutorial: Perl's Memory Management by oha (Friar) on Oct 28, 2009 at 13:36 UTC
I personally think that just givin a brief preamble on what are the differences between "freed" and "cleared" as for "end" and "exit" would make the document very usefull.	[reply]
Re^3: Mini-Tutorial: Perl's Memory Management by kyle (Abbot) on Oct 28, 2009 at 13:53 UTC
I agree, this would be much improved by some definitions. I read a lot of it thinking, "what is meant by 'free' here?". I think often times when questions about memory arise, a focus of interest is on when and how memory is returned to the operating system. That's probably a distinction made in here somewhere, but it would be better if that distinction were explicit.	[reply]
Re^2: Mini-Tutorial: Perl's Memory Management by Anonymous Monk on Oct 28, 2009 at 14:17 UTC
'free memory' is when you give up the memory block back to the OS, 'clear memory' is when you, for instance, say @a = ();, the variable still there but without content. as ikegami said, perl does not free memory, so look at free in C if you still don't understand..	[reply]
Re^3: Mini-Tutorial: Perl's Memory Management by oha (Friar) on Oct 28, 2009 at 17:21 UTC
for what i understood i disagree, there are 3 phases: clear release the bind of the data to the variable free the data is released to the mem manager so can be reused by perl free to OS may happen in some cases after the free. Where the first 2 are how perl behave, and the last depends on the system (OS + how perl was compiled) Am i wrong?	[reply]
Re: Mini-Tutorial: Perl's Memory Management by EvanCarroll (Chaplain) on Oct 27, 2009 at 22:38 UTC
Might also want to to add the reference to the perlfaq3. And, maybe mark up this post into perlfaq3. You can build perl to use your OS malloc pretty easily. I'm told Perl malloc is faster and more cross-platform, I would be curious to know why. You can find out what version your perl is using by `perl -V:usemymalloc`. If it is `n` as it is usually then you will never give back memory to the OS. I'd like to know why on Debian it is off. I assume it is the default build option, but why if gnumalloc is present? Evan Carroll The most respected person in the whole perl community. www.evancarroll.com	[reply]
Re^2: Mini-Tutorial: Perl's Memory Management by pid (Monk) on Oct 29, 2009 at 08:44 UTC
Based on my very own experience, Perl-malloc-enabled Perl does cause the crash of xchat2(with the Perl plugin) on FreeBSD, the problem seems to be unsolved yet. ref: http://www.freebsd.org/cgi/query-pr.cgi?pr=121472 I mean, sometimes, you have to usemymalloc but Perl's. :-p	[reply]
Re^3: Mini-Tutorial: Perl's Memory Management (which malloc) by tye (Sage) on Oct 29, 2009 at 17:31 UTC
As a link: http://www.freebsd.org/cgi/query-pr.cgi?pr=121472 If you have Perl use the system's default malloc() calls, then you can get sloppy and use Perl's configured free() on stuff that some library malloc()d or pass Perl-malloc()d buffers to an external library that expects to be able to free() or realloc() the buffer. So some xchat hacker got sloppy and didn't keep straight what needed to be Perl-allocated and what needed to be OS-allocated. - tye	[reply]
Re: Mini-Tutorial: Perl's Memory Management by Jenda (Abbot) on Oct 27, 2009 at 22:46 UTC
I'm not sure what is the intended audience of this article and the expected previous knowledge. As it is, it probably confuses more people than it helps. The fact that a variable (or rather the allocated structure) is not freed, but rather cleared and placed in some list of preallocated structures, is an implementation detail. A transparent optimization that 99.9% of people should not even know about. What they do need to know and be sure of is that the data will not be lost if there is a reference and that the memory will not be lost if there is no reference. Whether the part of memory goes to the general hash or some other place is not really their business. IMnsHO, there should be a big note on top that says something like "This is only for the curious, only for those that want to know what happens behind the scenes and is supposed to be hidden to the audience". and to tell the truth I don't understand what are you trying to say in the "undef does not free a variable" section. Jenda Enoch was right! Enjoy the last years of Rome.	[reply]
Re^2: Mini-Tutorial: Perl's Memory Management by ikegami (Patriarch) on Oct 27, 2009 at 23:09 UTC
The fact that a variable (or rather the allocated structure) is not freed, but rather cleared and placed in some list of preallocated structures, is an implementation detail. I said as much. ("This is an optimisation that should be transparent to you.") Misconceptions about the implementation details come up all the time. You've provided me with the perfect example in this paragraph. You believe it describes Perl when it doesn't. Where'd you get a list of preallocated structures from? Placing blocks of memory on a list for future use is how freeing a variable usually works. This is exactly what Perl doesn't do. I don't understand what are you trying to say in the "undef does not free a variable" section. The quote line contradicts what seems to be a popular belief. The section describes what `undef` actually does (frees the buffers associated with the variable).	[reply] [d/l]
Re^3: Mini-Tutorial: Perl's Memory Management by Jenda (Abbot) on Oct 28, 2009 at 10:06 UTC
"should" sounds a bit dangerous to me. "Should be transparent" ... or so it's not always? It can fail? Or what? How? The "should" seems to cause uncertainty. Mkay, let's see. If we declare the variable in a loop, we are about to do another iteration and there are no other references to the data, perl may clear the contents, but leave the structure (can't remember the name) and it doesn't have to allocate memory for that variable for the next iteration. Nice. Efficient. Now what does it do if it's not in any loop? Or it was the last iteration? What if it's a my variable inside a string eval""? I believe your tutorial will only make sense if you first explain at least a bit about how are the data stored (the variable name points to a SV which points to the string/contains the number/whatever) and then you can go on explaining what's "cleared" or "freed" and what does the Devel::Peek::Dump mean. Anyway for almost everyone the whole memory allocation tutorial should be "Declare variables with `my` in the tightest scope possible. Make sure you either do not create circular references, use weaken() to make sure at least one link in each circle is weak or break the circle by reseting at least one reference to undef. Do not expect perl to return the memory to the OS, but that doesn't mean the memory is not freed and will not be reused, it's just kept in perl's own hands. The rest is not your business." Jenda Enoch was right! Enjoy the last years of Rome.	[reply] [d/l]
Re^3: Mini-Tutorial: Perl's Memory Management by Anonymous Monk on Oct 28, 2009 at 02:17 UTC
I said as much. ("This is an optimisation that should be transparent to you.") That is very obtuse way	[reply]
Re^4: Mini-Tutorial: Perl's Memory Management by ikegami (Patriarch) on Oct 28, 2009 at 03:07 UTC
Re^5: Mini-Tutorial: Perl's Memory Management by Anonymous Monk on Oct 28, 2009 at 03:50 UTC
Re^2: Mini-Tutorial: Perl's Memory Management by jethro (Monsignor) on Oct 28, 2009 at 10:43 UTC
There are a lot of long-running perl scripts especially in the CGI area that depend on not using up more and more memory. But generally you have to do really dirty things to get bit by this. So the information is useful although probably not FAQ-worthy, especially not in this detail. And I had problems with understanding it on first reading too The most useful information in my opinion is "There is no guarantee that your perl version will give back memory to the system"	[reply]


Keep It Simple, Stupid
	PerlMonks