actually return memory to the OS. This is, in general, not possible. In *nix, once a process is allocated pages, there is no mechanism for giving them back.

I just checked this on linux (ubuntu 8.10 amd64) and can't confirm this statement

#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

main() {
    char *buf;
    char cmd[1024];

    snprintf(cmd, 1024, "ps -o pid,size,vsize h %d", getpid());
    system(cmd);
    buf = malloc( 1024*1024*1024 );
    system(cmd);
    free(buf);
    system(cmd);
    return 1;
}
[download]

And result:

10417   128   3780
10417 1048708 1052360
10417   128   3780
[download]

Update: and yes, it uses mmap