I just wonder if splitting the workload between two or more instances of your script will help. Whether this means updating a shared memory structure at the end of the processing, or similar (please forgive me if this doesn't make alot of sense - I have the flu and am at work).

For example, script #1 will process the first 10,000 lines, script #2 will process the second 10,000 lines, etc. then update a common structure at the end of the processing or parallel piped to a third script that merges the data

jason