in reply to Processing ~1 Trillion records
Can you give some background information of the problem at hand?
What database do you use? What does the query look like?
Restructuring a massive array and writing CSV files sounds costly. What pattern? How many CSV files?
Profiling a (small) subset of the data might give some indication. Did you time how long the various steps take?
In short: could you show some code please?