I'm no internals expert, but maybe you could sprinkle some print statements to print out your hash sizes, every now and then, and look for a pattern.

#!/usr/bin/perl

use Devel::Size 'total_size';

my %struct = ("key1" => ["bill","ben"],
             "key2" => ["dave","jen"]);

print "total size of %struct - ", total_size(\%struct),"\n";
[download]

I've been getting "bit" alot lately with "auto-vivication" of hashes. I would delete and undef a hash, and then it would get "recreated" accidently by some sub, which just tests for existing values. For instance:

#update last viewed timestamp  
     $info{$key_prev}{'timestamp'} = time;
[download]

Well I would delete and undef $info{$key_prev'}; and thought it was gone, but nope, it got auto-vivified. So I needed:

if(defined $info{$key_prev}){
   #update last viewed timestamp  
     $info{$key_prev}{'timestamp'} = time;
[download]

So maybe you should check for sneaky auto-vivications?