Current overview load all items into array of hash for each input item { complex transformation rules } save all items (timestamped) remove all items with old timestamp

Not commenting about the rest, but this part here will cost you both in speed and in memory. Any reason why you can't process each item seperately? If you avoid loading all your data into memory, your program will defenitly benefit:

 while (read next item) {
   complex transformation rules 
   save item unless it's too old
 }
[download]

-- Dan