a 'correct' (but possibly destructive) fix would be to rebalance the pages as new ones are inserted.

Just for clarification, when you say "rebalance the pages", are you referring to a process wherein the script recursively processes and merges the PDF files in "batches" of say 16 files a piece?

e.g.:

LEVEL 1 (4_096 files): [pdf_level1] [pdf_level1] [pdf_level1] ... [pdf
+_level1]
LEVEL 2 (  256 files): [[pdf_level1] * 16] [[pdf_level1] * 16] [[pdf_l
+evel1] * 16] ... [[pdf_level1] * 16]
LEVEL 3 (   16 files): [[pdf_level2] * 16] [[pdf_level2] * 16] [[pdf_l
+evel2] * 16] ... [[pdf_level2] * 16]
LEVEL 4 (    1  file): [[pdf_level3] * 16]
[download]