Perhaps you can make good use of Test::Memory::Cycle (and Devel::Cycle).

Can you reduce this to an isolated, stand-alone test case, using the minimum possible amount of code, that demonstrates the problem?   When tracking down memory leaks and reference-count problems, perhaps the problem is not strictly and totally in the code that you are now looking at.