Perhaps you have circular references somewhere that prevent structures from being garbage collected.

Try Devel::Leek and Devel::Cycle to find them.

Once you found them you can weak some of the references with Scalar::Util::weaken.