in reply to advice needed for processing largish data in multiple files

A hash with 1.5e6 keys requires around 300 MB if the value is a simple scalar, or roughly double that if the value is an array of 10 elements. Unless you are very low on memory, your initial instinct is probably a good one.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In Section Seekers of Perl Wisdom