Guessing at your data structure, you could use a line like this right after the inner loop:

It might work better if you were to use a tied hash that stored its contents on disk instead of in memory. That presumes you have enough disk space to hold your entries.

Do you need to build the complete data structure first or could you get away with File::Find callbacks to process files as they're encountered?