How much memory do you have? Your data structures are immense. It looks like you have perhaps 100,000,000 arrays in the hash, and even if most are not there you are autovivifying their keys ( scalar(@array) different anonymous hashes) and a reference to an empty array.

All those structures cost fifty-some bytes each plus a hefty multiple of the number of data elements.

I don't see a leak, just prolific use of memory. Can you read and use less of the data at a time?