a 'correct' (but possibly destructive) fix would be to rebalance the pages as new ones are inserted.
Just for clarification, when you say "rebalance the pages", are you referring to a process wherein the script recursively processes and merges the PDF files in "batches" of say 16 files a piece?
e.g.:
LEVEL 1 (4_096 files): [pdf_level1] [pdf_level1] [pdf_level1] ... [pdf
+_level1]
LEVEL 2 ( 256 files): [[pdf_level1] * 16] [[pdf_level1] * 16] [[pdf_l
+evel1] * 16] ... [[pdf_level1] * 16]
LEVEL 3 ( 16 files): [[pdf_level2] * 16] [[pdf_level2] * 16] [[pdf_l
+evel2] * 16] ... [[pdf_level2] * 16]
LEVEL 4 ( 1 file): [[pdf_level3] * 16]