Re^3: A danger of test driven development.

I was not attempting to critique TDD, I think the concept is a good one.

When I said "the problem with TDD" in my first sentence, I should have stuck with the OPs word, "the danger with TDD".

As I attempted to make clear in the second paragraph, "this ~~problem~~danger is not a fundamental problem with the methodology", or any other methodology, it is a problem of mis-application, or mis-emphasis, by some people, sometimes.

It is a matter of balance. If the balance swings to far in favour of one aspect of the development process, then others get watered down or totally omitted.

In the case of TDD, is easy for the programmer to write tests for the code they are going to write, instead of a test that encapsulates the specification. If their understanding of the specification is correct, then the code they intend to write will be correct and the tests will be correct.

But if their understanding of the spec is incorrect, then they write tests to test the code they intend to write, and from that point forth, code is verified against test and test against code, and the spec is forgotten until some third party attempts to use their code in conjunction with the spec.

Too many practitioners see the tests the programmer writes as the conclusion. They make great unit tests but very poor systems and integration tests.

The best test of an API, is to write an application that uses it. From the spec and preferably by someone other than writes the code that implements the API, or the person that specified it. But the latter is preferable to the former.

Having an application that uses the API is the surest way of ensuring that the API works for that application. Not just that individual functions and methods perform to requirements within the ranges of parameters that the application calls them on. But also, and perhaps more importantly, that the separate parts of the API work together and work with the rest of the application.

That is, not just "work" as in given parameters within specified ranges, produces the correct results, but work as in "is a good fit with the other functions within that particular API, and works with any and all other APIs and data representations that the application uses".

So you don't have one API that wants it data represented in relative format and another that takes absolute, forcing the application to constantly switch between formats when passing data between APIs. (Often as not, the one taking absolute format data will immediately convert it to relative format internally, or vice versa).

Eg. one uses DOB, and the age; or window coordinates versus screen coordinates; imperial units versus metric; and so on.

Once written, an application that uses an API will not only test for this type of impedance mismatch which is rarely tested by unit type testing (how many times have you used or read about APIs that "work fine but are a bitch to use"?). It will also tend to test the required range of parameters (and combinations thereof), of the individual APIs.

This will often tend to be a subset of the full range of possibilities that the API could be called upon to handle, but unless and until another application comes along that needs a wider set of (combination of) parameters. But if that second application never comes along, it can be an expensive exercise to write code and tests (or tests and code:) to handle them.

(Silly)eg. Writing a square root function to handle negative numbers and produce complex results would be an expensive option if it is never called upon to do so. Moreover, doing so could add further to the problems of the applications using it that would be better served by an exception being raised if passed a negative number than being returned a correct but useless imaginary result.

None of this means that TDD is a bad thing, only that over-emphasis of one aspect to the detriment of others is bad. "Oh! We'd never make that mistake", is everyones first reaction to the type of exaggerated examples I've cited, but they do happen.

Take the Mars Orbiter problem.

"This is an end-to-end process problem," he said. "A single error like this should not have caused the loss of Climate Orbiter. Something went wrong in our system processes in checks and balances that we have that should have caught this and fixed it."

Or the Short sighted Hubble telescope, the most perfectly flawed mirror ever produced up to that time.

Hubble is working perfectly but the Universe is all blurry.

TDD makes a virtue of having the programmer write the tests against which his code is tested. It works fine, provided that his myopic viewpoint is balanced by other techniques that give overview.

And that is the danger I am alluding to, not only with TDD, but any methodology that is given too high an importance to the exclusion of balance. If 100% test passed and 100% code coverage is not balanced by fitness for purpose, ease of use, good impedance matching with it's foreseen applications; if unit testing is not balanced against actual or "typical usage" systems and integration testing; then the overall result can be an expensive, but very well tested disaster.

It doesn't matter how many tests you write, or that they all pass, if you are testing the wrong thing.

If I have one fear of TDD, it is that it can lead to tunnel vision, unless there is someone or something within the project that has the authority and brief to take a much wider overview of the project, and how it will be used.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

Comment on Re^3: A danger of test driven development.

Replies are listed 'Best First'.
Re^4: A danger of test driven development. by Aristotle (Chancellor) on Oct 09, 2005 at 03:06 UTC
I know what you meant. What I’m saying is that this doesn’t strike me as a danger of TDD so much as a danger of not thinking. Whether you don’t think while following the TDD methodology or you don’t think while following any other methodology (including the no-methodology) really makes no difference – beyond determining in which way you end up screwing up. I suppose that implicit in your descriptions of the “danger” of TDD is that you intend to express “if you don’t think while following TDD, this is what is likely to happen.” I tend not to find such consequences very interesting because I tend to take thinking along as a given and a prerequisite, which makes your elaborations sound like they’re implying something more along the lines of “if you use TDD, you are likely to stop thinking.” Next to your statements showing you understand TDD quite well this creates an odd sort of cognitive dissonance, and that made me comment. If I’ve read you correctly, then now you also know how to read me. Makeshifts last the longest.	[reply]
Re^5: A danger of test driven development. by adrianh (Chancellor) on Oct 10, 2005 at 09:14 UTC
What I’m saying is that this doesn’t strike me as a danger of TDD so much as a danger of not thinking. Weeeell. Maybe. I think the difference testing code and testing specs is quite a subtle one - not one I would characterise as people not thinking. I have a requirement. I write a test to illustrate part of that requirement. I then write the minimal code I need to write to make that test pass - refactoring as I go. I have a requirement. I need to write some code to meet that requirement. I write a test that will fail without that code. I write the code. (1) is TDD. (2) is just writing the the test before your code. I've encountered a several people who say they're doing TDD when they're doing (2). It was certainly a stage I went through when I started down the TDD road. It's an easy trap to fall into. Well, more of a local maxima than a trap. (2) is still good - much better than not writing tests, or writing tests post-code. But it's not TDD because the design is not being driven by the tests, so you miss out on all the benefits that gives you. I agree this is not a danger of TDD. It's a danger of doing something that looks like TDD but isn't. Describing it as a danger of TDD is counter productive. People tend to read it as "I should not do FOO because BAR will happen" rather than "If BAR happens than I'm not doing FOO correctly". However I do find these consequences interesting. The places where people are doing wrong when they think they're doing right are useful areas to explore for at least two reasons. Firstly they highlight areas where people learning a practice can go wrong. One of the questions I regularly ask when I'm introducing people to TDD is "Why did you write that test?". If the answer is something like "Because the store can receive multiple orders at the same time" then things are probably going okay. If the answer is "Because I need to add an array to keep both orders" then some gentle nudging might be in order. Secondly they highlight areas where we might need additional mechanisms to catch the errors. For example the problem of the programmer tests and the spec getting radically out of sync doesn't tend to happen in groups doing Extreme Programming because they have additional practices (customer tests, not restricting TDD to just unit tests, short iterations, etc.) that catch these problems early or prevent them happening in the first place.	[reply]


Think about Loose Coupling
	PerlMonks