Regression testing is important. When writing Perl
modules with
ExtUtils::MakeMaker, it's usually quite
simple, as well. Personally, I often find writing these
tests to be both fun and reassuring.
However, like many people, I'm guilty of not writing tests
for Every Single Module I write or work on. Often when I
skip writing tests, it is because the module I'm working on
has external resource dependencies, which I don't think the
tests can always depend on, and/or I don't want them to use
for testing purposes. At best, this makes writing tests less than
simple. At worst, it can make writing tests more expensive
than just rewriting the module later when it's found to be
broken.
Some examples of resources a module may depend on are (by
"internal" I mean things the module programmers have
direct control over; external things are outsourced or
developed out-of-house):
- database servers
- other internal servers/services
- other internal modules with resource dependencies your
module shouldn't need to care about
- external modules with unknown dependencies
- external servers/services (Credit card fulfillment, for example)
- firewall status on the server your tests are running on
- environment: must be running under mod_perl/apache/etc. to work
We need to test the portions of the module which depend on these
resources (it might be every part of the module), or the module
isn't tested properly. But we can't always use the same
resources for testing as for production, which also makes
testing incomplete.
Making the assumption that your test scripts always have
access to these resources can be fatal:
- the resources may be unavailable to the test
script/server, causing the tests to fail. Sometimes this
means a resource needs to be installed; sometimes it means we've
firewalled it to protect it from test scripts :)
- the resources may be available, but set up in
production mode, and tests may break the production data
- externally controlled resources may not have a test mode
available
In these conditions, what do people typically do to build
useful test suites? I'm specifically thinking of the case
where it would be impossible or impractical to simulate
every resource dependency (such as when the amount of code
in the new module is small, but the level of resource
dependency is high: you'd spend more time simulating external
resources than in writing or testing the code).
I'm also interested in techniques people use to ensure that
only test resources are used while tests are being performed;
especially when using a (possibly externally controlled)
module with unknown resource dependencies.
Some solutions we've used, but which haven't provided a
complete answer yet, are:
- test databases with a different DSN in the test script
- propagate generic "test mode" flags to modules with known
dependencies. This isn't always doable for external modules.
- don't write tests when the test is known to be destructive
- use a "todo" test when it is likely to fail
- comment tests which may fail due to known resource unavailability
Any other ideas? I may have thought too much and mostly answered my own question, but I'm still interested in reading how others
have dealt with this sort of problem in the past. Thanks!
Alan
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
