aossama has asked for the wisdom of the Perl Monks concerning the following question:
Hello Monks,
I am in charge of re-factoring a script which basically iterates over an array from a SELECT query from a database which returns ~1 Trillion records, the array restructures the items in a certain pattern and produces csv files. The script works fine, but the issue I am facing is that it takes ~16 days to finish. I have to decrease the run-time of the script; So these were the thoughts I had:
I am in charge of re-factoring a script which basically iterates over an array from a SELECT query from a database which returns ~1 Trillion records, the array restructures the items in a certain pattern and produces csv files. The script works fine, but the issue I am facing is that it takes ~16 days to finish. I have to decrease the run-time of the script; So these were the thoughts I had:
- Grid Engine
- Hadoop
- Optimize the script
Back to
Seekers of Perl Wisdom