I'm in the process of introducing/improving automated testing in our environment, and looking at how we run and monitor those tests.

Background

We have a public facing web infrastructure based on Apache and mod perl, and a core customer facing client/server Java application. Development is done in a dev/test environment, unit tested by the developer there, then deployed into Integration where UAT testing happens. Finally it's deployed into production where end users break the use cases and script kiddies demonstrate that they have far less destructive imaginations than Joe User ;-) As usual, most of the testing for the website is manual, although we have an increasing number of TAP unit tests (the Java team have CI already - see below).

Continuous Integration

My initial thought was to run my 'runalltestswithdevelcover.sh' script once a day, redirecting the html output to a webserver. I've got that basically set up, and it looks like a good start. It outputs some nicely HTML formatted TAP output courtesy of TAP::Formatter::HTML and Devel::Cover output. I'm also contemplating a spot of integration with CVS to show when tested files were last checked in as well.

But then one of our Java team mentioned continuous integration to me. I had a bit of a poke around in TeamCity (which the Java team use) and concluded that it would be really nice to integrate a CI system with our build process. The way I'm thinking, the overnight process becomes:

1. Run build script
2. deploy daily build to (an) integration server
3. run unit tests and report output
4. back out build
5. run previous unit test set and report output

That would test not only the code, but the release/deployment process (subject to the limitations of automated tests of course) and the backout process too.

Incidentally this would probably entail a second integration server. The deployment process entails process controls on the integration environment, which is widely used outside our team so we can't risk regularly breaking it.

It would be a fairly time consuming and complex task to build this testing & CI infrastructure, and my initial idea is to do it gradually. The change plan would be:

• Stage 1: a daily run in Dev of the existing unit test suite, outputting the results to TeamCity
• Stage 2: build a 2nd integration server/VM and build the scripts to keep it in sync with the main integration server
• Stage 3: get TeamCity running the build script daily, to ensure the build is successful
• Stage 4: start deploying the build to the second integration server daily and recording the results
• Stage 5: Before and after unit testing in integration as part of the process

To me it looks do-able, if a little time consuming, and likely to improve our release process significantly. I'd quite like to make the right choices before I start, so that I don't end up wasting a load of time on tools that won't do what I want. So:

• Has anyone else built this kind of release infrastructure for a LAMP environment?
• Will TeamCity work ok for this, or should I be looking at Smolder, or Hudson, or something else?
• Anyone got any other suggestions/thoughts?

Thanks.

Charles.

Notes:

Suggestions on Deploying Perl Test Environment

TeamCity on CPAN