Generally speaking, when I've had that problem, it was that there was a memory limit via ulimit - removing that limit 'solved' the problem. At least, insofaras letting me use way more memory. Perhaps you have an underlying memory problem - wasting memory or leaking it, we can't be sure from your description. Assuming that not to be the case, though, it's probably a ulimit on memory.