This is something you really want to solve at the OS
level, and not at in the program itself. One way to solve
this is by setting ulimits, and limit the
allowable memory a process can use.
If you happen to have glance you could also
write an advisor script to handle situations like this.