in reply to Memory issue with large array comparison
This looks like a pretty standard "scan one list for items existing (or not existing) in another list" problem, which screams for a hash. Make a hash using the values from your second array (like GS12345) as the keys. Then loop through your array of filenames, use split or a filename module to break out the ID from the pathname and see if it exists as a key in the hash.
With arrays that large, memory shouldn't be a problem doing it that way. I just created a hash with 10,000 7-character keys, and Devel::Size says it only used 1MB of memory. If your array of pathnames is running you out of memory, maybe you can process them one-by-one from wherever they're coming from, instead of loading them all into an array at once.
Aaron B.
Available for small or large Perl jobs; see my home node.
|
|---|