Maybe your "unix toolbox" (i assume you use a *nix system?) could help, to figure out what components of your program use how much memory (pmap -q processid), lsof e.g.
You could also tune the "swapiness" of your system to free some memory, Watch how much memory is allocated through shared libraries e.g.
a google search should help to find the right parameters), eventually your programm could be optimised too, but thats beyond my experience ...