Re: demonstrate that perl can give back memory to the OS

Define 'give back'... I can't speak to Windows, but for every flavor of *nix I have ever used:

make it re-usable by the current process. There are many situations where perl does this
actually return memory to the OS. This is, in general, not possible. In *nix, once a process is allocated pages, there is no mechanism for giving them back. The only exception to this is memory added via mmap. If you munmap a file, and you are the only process that had it mapped, then this memory is returned to the OS. mmap, however, is typically only used for very specific things, like shared-lib access
'Kind of give it back'. If the memory you are no longer using spans pages, and you have a lot of memory in use, then the system will swap those pages out. They are still considered used by the OS, but they will never actually be put back into RAM. In most situations, this will not occur, because almost nobody wants to run a system that actually has to swap, since it usually comes with a significant performance hit.

update There are definitely situations where malloc does indeed make use of mmap instead of sbrk to obtain memory from the OS, for Linux and BSD. I believe that this is limited to single large allocations (>1MB by default), where there is not already that much contiguous free within the heap.

Comment on Re: demonstrate that perl can give back memory to the OS

Replies are listed 'Best First'.
Re^2: demonstrate that perl can give back memory to the OS by almut (Canon) on Feb 27, 2009 at 18:53 UTC
2. actually return memory to the OS. This is, in general, not possible. In *nix, once a process is allocated pages, there is no mechanism for giving them back. The only exception to this is memory added via mmap. (...) mmap, however, is typically only used for very specific things, like shared-lib access But this seems to be just what is happening (i.e. mmap/munmap): `PID SZ VSZ 5645 604 11064 PID SZ VSZ 5645 4516 14976 PID SZ VSZ 5645 2560 13020` [download] (This is the output I get when running Joost's snippet — even on an old Linux box with a rather rusty 2.6.16 kernel.) The last memory printout is definately about 2M smaller (the PV/string part of that scalar variable) than the previous one... And `strace` shows — when I replace the `system("/bin/ps",...)` with a simple `print "---\n"` (to leave a marker in the output): $ strace ./746953.pl (...) brk(0x66a000) = 0x66a000 read(4, "#!/usr/bin/perl\n\nsub ps {\n #s"..., 8192) = 1003 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 write(1, "---\n", 4--- ) = 4 mmap(NULL, 2002944, PROT_READ\|PROT_WRITE, MAP_PRIVATE\|MAP_ANONYMOUS, - +1, 0) = 0x2ba8bbdd5000 mmap(NULL, 2002944, PROT_READ\|PROT_WRITE, MAP_PRIVATE\|MAP_ANONYMOUS, - +1, 0) = 0x2ba8bbfbe000 write(1, "---\n", 4--- ) = 4 munmap(0x2ba8bbfbe000, 2002944) = 0 write(1, "---\n", 4--- ) = 4 lseek(4, 196, SEEK_SET) = 196 lseek(4, 0, SEEK_CUR) = 196 close(4) = 0 exit_group(0) [download]	[reply] [d/l] [select]
Re^3: demonstrate that perl can give back memory to the OS by Illuminatus (Curate) on Feb 27, 2009 at 19:10 UTC
For shared-mem/mmap operations, you can't rely on ps, as every process that mmap's a file will have it included in it's total memory usage. There is really only a single copy in RAM for the OS -- it is simply mapped into different locations of different processes. When a given process munmaps a file, its 'ps' memory usage will say it is smaller, but the memory is still in use in the other processes. Only if it is the last process to have the file mapped will the memory actually be 'returned' to the OS. And, again, mmap is generally only used for shared libraries.	[reply]
Re^4: demonstrate that perl can give back memory to the OS by almut (Canon) on Feb 27, 2009 at 19:53 UTC
For shared-mem/mmap operations... What you're saying about shared mmap is correct, but I'm not sure why you think this has to do with what's going on here. The strace clearly shows MAP_PRIVATE, and also, why should that string (`$x`) be shared with other processes (like code of shared libs)? And, again, mmap is generally only used for shared libraries. I don't think so (though it depends on how you define "generally"). The three `write` lines in the log mark: (1) before allocation and (2) after allocation of `my $x = " " x 2_000_000;`, and (3) after `undef $x;`. Why would those mmap/munmaps that happen in between those markers be related to loading of shared libs, or some such? Doesn't make much sense to me. The only thing that remains to be explained (IMO) is why perl mmaps two blocks of 2002944 bytes, but only munmaps one... (When you do the same with an array, e.g. `my @x = (); $x[1_000_000] = 0; # mmaps 8003584 bytes undef @x; # munmaps 8003584 bytes` [download] the required size (to grow the array) is mmapped/munmapped exactly once each.)	[reply] [d/l] [select]
Re^4: demonstrate that perl can give back memory to the OS by zwon (Abbot) on Feb 27, 2009 at 19:30 UTC
mmap is generally only used for shared libraries AFAIK, modern malloc(2) implementations actively use mmap to allocate memory, so it's not correct	[reply]
Re^2: demonstrate that perl can give back memory to the OS by zwon (Abbot) on Feb 27, 2009 at 19:05 UTC
actually return memory to the OS. This is, in general, not possible. In nix, once a process is allocated pages, there is no mechanism for giving them back.* I just checked this on linux (ubuntu 8.10 amd64) and can't confirm this statement `#include <sys/types.h> #include <unistd.h> #include <stdlib.h> #include <stdio.h> main() { char buf; char cmd[1024]; snprintf(cmd, 1024, "ps -o pid,size,vsize h %d", getpid()); system(cmd); buf = malloc( 102410241024 ); system(cmd); free(buf); system(cmd); return 1; }` [download] And result: `10417 128 3780 10417 1048708 1052360 10417 128 3780` [download] Update:* and yes, it uses mmap	[reply] [d/l] [select]