I've built applications with similar memory requirements. Generally there is no reason to store all this in RAM unless you are performing complex matrix operations from the top to the bottom. I would refactor the code and so that it processes one chunk/one line/one customer (whatever) at a time and writes to a file. Better still, if you can, generalise the transformation into code units, you can then string them together with pipes and let the OS worry about disk and RAM management.