G'day Martin90,

There's not really enough information to say whether those steps will overload your system; for instance, you don't provide any information about your system nor do you indicate what "Analyze stored keys" involves. I also don't understand why you suggest "slow things a liittle bit (sleep?)" when you've already indicated a set timeframe ("every hour") for processing.

Having said that, here's some pointers that might help:

• Keep track of each stage (possibly a simple text file indicating progress); include code to allow restarting from some known good point.
• Include "Step 2" in the 'while (<$fh>) {...}' loop.
• Use Storable to serialise the hash prior to performing whatever analysis is required.
• It's a decade since I last used MySQL — I don't know what facilities are available but you should make the database update all or nothing.
• You can speed up your regex by anchoring the pattern to one, or both, ends of the string if possible (e.g. with '^', '$', etc.) and not using the 'i' modifier.