I'm still going to finish my file mapped attempt.

For a 100GB file, using the tradition method, you need to read the original file; write the spill files; read the spill files; write the final file. That's 400GB of reads and writes at 61MB/s; that's a bit under 2 hours (1.865) of IO.

If I can do it "in-place"; half of that (56 minutes) is up for grabs.

Given that I can swap the two halves of a 4GB buffer in ram in 1.177196260s, that gives the potential to move around 12TB of data around in the same time as it takes just to write and re-read the spill files.

Of course, that's optimistic because mem-mapped data gets written back to disk at some point; but given the 3 layers of cache between ram and disk, with the right choice of algorithm(s), and the 3072 times headroom; there ought to be the possibility to save a substantial chunk of time.

If you view the mem-mapped file with its set of sorted sub-partitions as the spill files; you only need access to the top of each partition if your doing an N-way merge. So if you have a minimal mapped-view (say 64k) at the top of each of 50 2GB sorted sub-partitions, that's a bit over 3MB, leaving the rest of memory for use as the 'output file'. It does mean writing the accumulated output to a separate file (in chunks), but maybe that's okay.

Another possibility is to move the first 2GB buffer into unmapped ram, run the N-way merge, writing the output over the freed up mem-mapped first buffer. Once that is full, move the (remainder of) the second buffer into unmapped ram and repeat until done.

Now take that a step further: Instead of copying the whole of the first 2GB buffer into unmapped ram, do it in 64k or 1mb chunks as needed to free space for the N-way to write to.

Just a bunch of random thoughts at the moment...