in reply to Performance Question

While I have no direct wisdom to share, I will tell you which 2 questions immidiately came to mind when I read your post. Maybe they will shed some lite:

  1. You said: "monitoring the output, it looks like it will take 166 hours to run What are you baseing this number on? How are you doing this monitoring? are you sure that your method of monitoring the output file isn't flawed?
    One scenario I can easily imagine is if you are just checking the file's size X seconds after starting the program, dividing into 81GB, and multiplying by X. unless you turned on autoflush, maybe you just happened to check the file's size just before it was about to do a batch write -- completely skewing your estimate.

  2. Where is your code? A perl program like you describe sounds extremely simple, but whenever people post questions about programs that do things "simply" without posting any code, I tend to wonder what else is going on. What are "any needed changes" ? are you sure there isn't something else you are doing that's taking a lot of time?