This sounds like it is a one-off.

Assuming that you don't need to maintain state (ie remember what has come before in the file) and that you are processing each line independently, it may be easier to just divide the file into (eg) 5, and run the same script on each sub-file, on separate computers if need be.

It'd certainly be a lot simpler than debugging a threaded application.