How much disk space do you have available to you? Would it be possible to split each file into a number of temporary files, then process each of those files in turn, then concatenate them when all of the temp files have been processed?