in reply to need to optimize my sub routine
I don't know about windows, but on unix/linux/macosx, you can run "top" in a separate terminal while your process is active, and watch what happens in terms of memory consumption and page faults, in addition to overall cpu load.
Perl data structures (like your AoHoH "@data") take up a lot more space than you might expect. 120 MB of data in disk files might take up two or three (or more?) times that amount inside the perl process, because of all the overhead associated with managing scalar values and nested structures.
So figure out whether you really need all your file data to be (virtual) memory resident at the same time in that one huge AoHoH structure. Maybe only some of the data from each csv row needs to be kept, or maybe the processing can be done serially (i.e. while reading each file)?
|
|---|