Agreed, with one exception - that being if you want to re-factor the code or make major architectural changes. Then you need tests that exercise the application thoroughly so that you can be sure that your changes don't break anything.

As for how to write an application in such a way as to be easily tested - instead of a huge monolith, break it into chunks so that each little bit of functionality can be easily tested. For example, if your application has a config file, then instead of scattering configgish stuff throughout the app, abstract it out into a module that provides functions for reading and parsing the file, and for getting at the information, which can be tested entirely seperate from the main body of the app.