Some quick testing by simply changing from Array of Hash to Array of Array, and using Top to compare total process mem:

Input data: 40MB

Array of 67,000 Hash: 350MB

Array of 67,000 Array: 190MB

So there appears to be a reasonable memory saving to be had. My biggest production datasets are 7 to 10 times larger - what process memory usage this will translate into is hard to anticipate, and difficult to test at the moment, but I would make working assumptions of 2G vs 3G - and I would rather the 160M - 1G was used for the DB cache than the Perl Hash.

Regards,

Jeff