Look for circular references, meaning: the children holding a reference to their parent, themselves, each other, .... You seem to be describing the typical symptoms.

If you find them, weaken these references with weaken() from either WeakRef or from Scalar::Util, depending on the version and compilation status of the latter — the pure perl version won't contain it. That way, the backreferences won't keep the object they're referring to, alive. Make sure you don't weaken the important, forward references.