Presumably you have circular references.  In short: Perl's garbage collection mechanism is based on counting references to variables/objects. When some data structure is referring back to itself, the reference count doesn't drop to zero when it goes out of scope (or is being undef'ed), and the memory isn't being freed.

There are a couple of modules for analysing and getting rid of the problem, e.g. Devel::Leak, Devel::LeakTrace, Devel::Cycle, Object::Destroyer, Devel::Monitor. (The docs of the latter module contain a rather elaborate explanation of the problem, btw.)