A completely unsatisfactory (in terms of self image anyway) but completely practical solution is the restart the server from time to time. You can even have the server restart itself.

How much time is it worth spending trying to hunt down this issue? Put a restart procedure in place, possibly triggered by high memory use, then drop the priority for finding a solution down to a level that doesn't get in the way of productive work.