The testing time alone to re-factor would be large

Ahh...a lesson for next time. If you have good unit tests from the start then you can refactor all you like; your unit tests will tell you if you've broken something. This is one of the tenets of Extreme Programming and for me was an "aha!" moment when I fully grok'd why it was so important.