Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

demonstrate that perl can give back memory to the OS

by perl5ever (Pilgrim)
on Feb 27, 2009 at 16:58 UTC ( [id://746953]=perlquestion: print w/replies, xml ) Need Help??

perl5ever has asked for the wisdom of the Perl Monks concerning the following question:

I've heard that if the conditions are right (OS, perl version, C library, etc.) perl can give back free memory to the OS.

1) Does anyone know of a set of conditions under which this will happen?

2) Can someone provide a simple demonstration script which shows that this is happening? I'm trying something like:

sub ps { system("/bin/ps", "-o", "pid,size,vsize", $$); } ps(); { my $x = " " x 2_000_000; ps(); } ps();

Thanks!

Replies are listed 'Best First'.
Re: demonstrate that perl can give back memory to the OS
by Joost (Canon) on Feb 27, 2009 at 17:05 UTC

      An explanation:

      Lexical (my) variables and their string (or array) buffers aren't deallocated on scope exit whenever possible, only cleared. undef $var; (but not $var = undef;) deallocates the variable's string buffer.

Re: demonstrate that perl can give back memory to the OS
by BrowserUk (Patriarch) on Feb 27, 2009 at 20:06 UTC

    The breakpoint on Win23/AS5.8.6 seems to be that for any allocation over 64k, a separate request is made to the OS (VirtualAlloc()) to fulfill that allocation rather than allocating from the existing heap. And if that entire allocation is released in one go, then it will be released back to the OS memory pool.

    YMMV on other setups.

    This can be verified by doing:

    $x = chr(0); $x x= 2**16; undef $x;

    And noting the MemUsage as displayed in the Task Mananger or PrcessExplorer after each step.

    A screen grab. Note the 64k bump in the middle trace.)


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: demonstrate that perl can give back memory to the OS
by jettero (Monsignor) on Feb 27, 2009 at 17:08 UTC
    I think, even if you give the memory back to the OS, it won't necessarily claim it until it needs it. There's no way to make the OS take it back and ps won't reflect the fact that the memory is released until it's reclaimed.

    -Paul

Re: demonstrate that perl can give back memory to the OS
by Illuminatus (Curate) on Feb 27, 2009 at 17:57 UTC
    Define 'give back'... I can't speak to Windows, but for every flavor of *nix I have ever used:
    1. make it re-usable by the current process. There are many situations where perl does this
    2. actually return memory to the OS. This is, in general, not possible. In *nix, once a process is allocated pages, there is no mechanism for giving them back. The only exception to this is memory added via mmap. If you munmap a file, and you are the only process that had it mapped, then this memory is returned to the OS. mmap, however, is typically only used for very specific things, like shared-lib access
    3. 'Kind of give it back'. If the memory you are no longer using spans pages, and you have a lot of memory in use, then the system will swap those pages out. They are still considered used by the OS, but they will never actually be put back into RAM. In most situations, this will not occur, because almost nobody wants to run a system that actually has to swap, since it usually comes with a significant performance hit.
    update There are definitely situations where malloc does indeed make use of mmap instead of sbrk to obtain memory from the OS, for Linux and BSD. I believe that this is limited to single large allocations (>1MB by default), where there is not already that much contiguous free within the heap.
      2. actually return memory to the OS. This is, in general, not possible. In *nix, once a process is allocated pages, there is no mechanism for giving them back. The only exception to this is memory added via mmap. (...) mmap, however, is typically only used for very specific things, like shared-lib access

      But this seems to be just what is happening (i.e. mmap/munmap):

      PID SZ VSZ 5645 604 11064 PID SZ VSZ 5645 4516 14976 PID SZ VSZ 5645 2560 13020

      (This is the output I get when running Joost's snippet — even on an old Linux box with a rather rusty 2.6.16 kernel.)

      The last memory printout is definately about 2M smaller (the PV/string part of that scalar variable) than the previous one...

      And strace shows — when I replace the system("/bin/ps",...) with a simple print "---\n" (to leave a marker in the output):

      $ strace ./746953.pl (...) brk(0x66a000) = 0x66a000 read(4, "#!/usr/bin/perl\n\nsub ps {\n #s"..., 8192) = 1003 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 write(1, "---\n", 4--- ) = 4 mmap(NULL, 2002944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, - +1, 0) = 0x2ba8bbdd5000 mmap(NULL, 2002944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, - +1, 0) = 0x2ba8bbfbe000 write(1, "---\n", 4--- ) = 4 munmap(0x2ba8bbfbe000, 2002944) = 0 write(1, "---\n", 4--- ) = 4 lseek(4, 196, SEEK_SET) = 196 lseek(4, 0, SEEK_CUR) = 196 close(4) = 0 exit_group(0)
        For shared-mem/mmap operations, you can't rely on ps, as every process that mmap's a file will have it included in it's total memory usage. There is really only a single copy in RAM for the OS -- it is simply mapped into different locations of different processes. When a given process munmaps a file, its 'ps' memory usage will say it is smaller, but the memory is still in use in the other processes. Only if it is the last process to have the file mapped will the memory actually be 'returned' to the OS. And, again, mmap is generally only used for shared libraries.
      actually return memory to the OS. This is, in general, not possible. In *nix, once a process is allocated pages, there is no mechanism for giving them back.

      I just checked this on linux (ubuntu 8.10 amd64) and can't confirm this statement

      #include <sys/types.h> #include <unistd.h> #include <stdlib.h> #include <stdio.h> main() { char *buf; char cmd[1024]; snprintf(cmd, 1024, "ps -o pid,size,vsize h %d", getpid()); system(cmd); buf = malloc( 1024*1024*1024 ); system(cmd); free(buf); system(cmd); return 1; }

      And result:

      10417 128 3780 10417 1048708 1052360 10417 128 3780

      Update: and yes, it uses mmap

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://746953]
Approved by Joost
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others learning in the Monastery: (4)
As of 2024-04-26 04:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found