Whilst the data originates on disk and is bigger than memory; the two buffers being merged here are both fully in memory, but combined are close to the limits of memory, hence not enough space to perform the n-way merge.

If the overall pair of data sets are bigger than memory, thus requiring doing it in chunks that fit in memory, why not smaller chunks?

why not smaller chunks?

Because eventually, you need to merge the smaller chunks into bigger ones. That includes the ones bigger than memory, but by spltting into 1/2 memory sized chunks, you can merge them in pairs:

A B C D        Each 2GB
A&B
  B&C
    C&D        The largest have migrated from A to D, no need to revis
+it
  B&C
A&B            The smallest have migrated from D to A no need to revis
+it
  B&C          And final pass ensures everything is in place.
[download]

Using smaller buffers only delays the inevitable and increases the number of passes.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

I understand that, but I still wonder if the performance boost afforded by simplifying the "shuffling around" could more than offset the extra overhead of more merges.