If you require that the entire file be held in memory at once, create a supervisor script that assigns a file to a separate worker process and waits for results. When the worker process finishes and sends its results back to the supervisor, the worker terminates, freeing its resources. The supervisor's memory consumption will remain stable at all times. The worker processes may use a little, or a lot... but when they're done, they vanish and release their memory.