Why don't you print the output to several smaller files? Depending on what you need the data for and wat you're going to with it, this would shorten the time until the user gets some output, and you might even consolidate the files again for later use.

Note that this is just off the top of my hat, not really thought through and maybe completely useless.