Computer programs, these days, ought to be fast. We have “an embarrassment of riches” when it comes to hardware and speed. So, if your program isn't performing as fast as you think it should, take a close look at your algorithm. Look for hidden sources of inefficiency ... and for inefficiency, look for I/O.

This opinion is highly biaised by the kind of problems you usually seem to solve. In scientifical computing, most problems don't involve I/O at all and the algorithms are often so straightforward that it'll never be possible to improve them. In those case, it's the sheer power of the CPUs that talks and the microoptimizations (the very same that receive so much hate from the so-called "clean" programmers) that makes you win hours, days or weeks.