I don't know exactly why you mention MacOS... by which I assume you mean pre-OSX. On Macintosh the programs not only can't give back memory to the OS, they can't even receive more memory if they need it. All memory is preallocated to a program when it starts up, and that's all it can get. The Finder (the GUI layer on top of the OS) is the only program not restricted by this rule. AFAIK.