Yes. I try to work that way as often as I can. By writing the tests I get to play with the API before I commit to anything in code -- in essence, I refactor the API using the tests as a "prototype".

The other great thing is that it forces oneself to be specific as to what one expects often makes the coding job simpler.

Plus, I get to check that the tests actually *fail* when the code doesn't exist. When the code already is there and you write a test and it passes, how do you know your tests is actually doing what you think? Do you go back and break your code to confirm your tests -- if you're like most people, probably not. But by writing the test first, you get to see it both fail and then succeed, which increases the robustness of the test as well as the code.

Where I tend to make exceptions:

• Some things are just really hard to write simple tests for (interactivity, complex dependencies on external programs or systems, etc.) If the process of writing a test means significantly more work than the code itself, and I'm pretty sure I can manually test some core cases, then I'll occasionally cut corners.

  That said, mocking complexity is often your friend in this process. And even things that seem complex can be tested. My recent challenge-to-self is to have a smoke tester be able to test its own smoke testing ability. C.f. CPAN::Reporter::Smoker. It actually bundles a tiny mini-cpan inside the test suite for testing.

• Some "defensive" coding and exception coding (invalid value testing). Often, things like "open or die". In my opinion, a pedantic focus on test-first shouldn't get in the way of good practices and momentum. I don't want to write a test to fake a file that can't be opened just to let me "test" before I write the "or die" part. Running coverage tests periodically with Devel::Cover is important to keep me honest about going back and writing tests later for those things I just threw in when I was on a roll.

Not me. Whilst the 'tests first' brigade are writing tests to see if Perl remembers how to load a module, detect a non-existent or privileged file, and hasn't forgotten how to do math, I'll be writing an* application that uses the module.

Functions/methods within that module will simply return reasonable constant data/predefined status until the application is written, compiles and runs. Once the application runs, then I start filling in the bodies of of those APIs one by one, checking the application continues to run correctly as I go. Adding asserts in the APIs to check parameters. And asserts in the application to check returns.

I find it infinitely preferable to have the application or module die at a named line, of a named file, so that I can go directly to the failing assertion, than to

• sit and watch streams of numbers, dots and percentages scroll off the top of my screen.
• And then have to re-run the tests and pause/abort it to find the name of a .t file and some imaginary number.
• Then go find that .t file and try and count my way through the tests to find which one failed.
• Then try and work out what output the test actually produced that didn't match the expected output.
• Then try and relate that to something in the module that failed to work as expected.
• Then go find the appropriate line of the module.

And that's the abbreviated version of many of the test failures I've had reported by modules using the Test::* philosophy.

A test should be located as close as possible to the point of failure--and tell me what went wrong and where. Anything else is just mastication.

*An, not (necessarily) the application. A contrived and simplified sample that exercises the full API, is fine. It can also act as user documentation/sample code.

• sit and watch streams of numbers, dots and percentages scroll off the top of my screen.
• And then have to re-run the tests and pause/abort it to find the name of a .t file and some imaginary number.
• Then go find that .t file and try and count my way through the tests to find which one failed.
• Then try and work out what output the test actually produced that didn't match the expected output.
• Then try and relate that to something in the module that failed to work as expected.
• Then go find the appropriate line of the module.

I respect the point you're trying to make, but I think the issues you present above can be addressed by greater granularity in the test-code cycle. I never write a complete test file and then write code to meet it. That's just asking for unnecessary pain.

Instead, before I write a subroutine, I think of how I'll know that I coded it correctly. What inputs does it take? What output should it provide? Then I go write a test for just that piece.

When I test just that *.t file (e.g. with "prove"), I should see any existing tests pass and that one test I just wrote fail.

Then I write the code to make that test pass. Then I prove that *.t file again -- if the test passes, great. If not, then either the code is wrong or the test is wrong -- and sometimes it's the test (hey, it's just code and bugs can show up there, too.) It's like double-entry accounting.

So I don't generally have to go through all the (legitimately annoying) test output analysis you describe. I know what test I just wrote and I know what code I just wrote.

To the point about "counting tests", test failures should be reporting where in the *.t file the test. That said, I often code with data structures that loop over test cases -- and so I write defensively with good labels for each case that let me quickly find the failing case.

Only periodically, after I've finished a substantial chunk of work, do I run the full test suite to make sure I haven't inadvertently broken things elsewhere.

As with everything, you make a great point about avoiding pedantry. Is is really necessary to test that a file loads? Or that math is done correctly? No, of course not. What's that line about 'foolish consistency'?

That said, the cost of writing a line of test for trivial code is pretty minimal, so I often think it's worth it since what starts out trivial (e.g. addition) sometimes blossoms over time into function calls, edge cases, etc. and having the test for the prior behavior is like a safety net in case the trival becomes more complex.

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

When I test just that *.t file (e.g. with "prove"), I should see any existing tests pass and that one test I just wrote fail.

That's fine when you're writing a given piece of code. You know what you just added or changed and it's probably sitting in your editor at just the right line. With the minuscule granularity of some of the test scripts I've seen, a failing test probably translates to an error in a single line or even sub clause thereof.

But most of my interactions with Test::* (as someone who doesn't use them), are as maintenance programmer on newly built or installed modules running make test.

I've not just forgotten how the code is structured, and how the (arbitrarily named) test files relate to the structure of the code. I never knew either.

All I see is:

...
[gobs of useless crap omitted]
...
t\accessors.......ok 28/37# Looks like you failed 1 test of 37.
t\accessors.......dubious
        Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 14
        Failed 1/37 tests, 97.30% okay
...
[gobs more crap omitted]
...
[download]

• Something failed.
• 28 from 37 == 9. But it says: "# Looks like you failed 1 test of 37."
• Then it usefully says: "dubious". No shit Sherlock.
• Then "Test returned status 1 (wstat 256, 0x100)". Does that mean anything? To anyone?
• Then (finally) something useful: "DIED. FAILED test 14".

Now all I gotta do is:

1. work out which test--and they're frequently far from easy to count--is test 14.
2. Where in accessors.t it is located.
3. Then work out what API(s) that code is calling.
4. And what parameters it is passing.

   Which if they are constants is fine, but if the test writer has been clever efficient and structured a set of similar tests to loop over some data structure, I've a problem.

   I can't easily drop into the debugger, or even add a few trace statements to display the parameters as the tests loop, cos that output would get thrown away.

And at this point, all I've done is find the test that failed. Not the code.

The API that is called may be directly responsible, but I still don't know for sure what file it is in?

And it can easily be that the called API is calling some other API(s) internally.

• And they can be located elsewhere in the same file.
• Or in another file used by that file and called directly.
• Or in another file called through 1 or more levels of inheritance.

And maybe the test writer has added informational messages that identify specific tests. And maybe they haven't. If they have, they may be unique, hard coded constants. Or they could be runtime generated in the test file and so unsearchable.

And even if they are searchable, they are a piss poor substitute for a simple bloody line number. And they required additional discipline and effort on the behalf of someone who I've never met and does not work for my organisation.

Line numbers and traceback are free, self maintaining, always available, and unique.

If tests are located near the code they test, when the code fails to compile or fails an assertion, the information takes me directly to the point of failure. All the numbering of tests, and labelling of tests, is just a very poor substitute and costly extra effort to write and maintain--if either or both is actually done at all.

That said, the cost of writing a line of test for trivial code is pretty minimal, so I often think it's worth it since what starts out trivial (e.g. addition) sometimes blossoms over time into function calls, edge cases, etc. and having the test for the prior behavior is like a safety net in case the trivial becomes more complex.

This is a prime example of the former of the two greatest evils in software development today: What-if pessimism and Wouldn't-it-be-nice-if optimism.. Writing extra code (especially non-production code) now, "in case it might become useful later" costs in both up-front costs and ongoing maintenance.

And sods law (as well as my own personal experience), suggests that the code you think to write now "just in case" is never actually used. Though inevitably the piece that you didn't think to write, is needed.

Some code needs to cover all the bases, consider every possible contingency. If your code is zooming around a few million miles away at the other end of a low-speed data-link burned into eprom. Then, belt & braces--or even three belts, two sets of braces and a reinforced safety harness--may be legitimate. But very few of us, and a very small amount of the world's code base live in such environments.

For the most part, the simplest way to improve the ROI of software, is to write less of it! And target what you must write, in those areas where it does most good.

Speculative, defensive, non-production crutches to future possibilities will rarely if ever be exercised, and almost never produce a ROI. And code that doesn't contribute to the ROI is not just wasted capital investment, but an ongoing drain on maintenance budgets and maintenance team mind-space.

Far better to expend your time making it easy to locate and correct real bugs that crop up during systems integration and beta testing, than trying to predict future developments and failure modes. And the single best contribution a test writer can make to that goal is to get the maintenance programmer as close to the source of the failure, when it occurs, as quickly as possible.

Yes. It is possible to speculate about what erroneous parameters might be fed to an API at some point in the future. And it is possible to write a test to pass those values to the code and ensure that the embedded range checks will identify them. But it is also possible to speculate about the earth being destroyed by a meteorite. How are you going to test that? And what would you do if you successfully detected it?

And yes, those are rhetorical questions about an extreme scenario, and there is line somewhere between what is a reasonable speculative possibility and the extremely unlikely. But that line is far lower down the scale than most people think.

Finally, it is a proven and incontestable fact that the single, simplest, cheapest, most effective way to avoid bugs is to write less code. And tests are code. Testing is a science and tests should be designed, not hacked together as after-thoughts (or pre-thoughts).

Better to have 10 well designed and targetted tests, than 100 overlapping, redundant, what-ifs.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

First, I appreciate the dissenting opinion, I was looking particularly for the proverbial "other ways to do it."

I suspect that the experence difference between you and I is the main point here: your method works best for you because you have the skill to avoid the pitfalls. Were I to design something in that way, I'm sure that I'd miss an assertion or return check somewhere, or the coverage wouldn't be complete in some other way. The Test::* method allows me to have all my tests in the same place, where gaps are obvious. The extra work initially has a big payoff downstream when I'm hunting bugs.

A few points here:
• Yes, what gets reported in a failing test could be seriously improved. I have fully drunk the TDD kool-aid and I hate the results when I have a failing test in a .t file with 523 test cases. Often, I have to throw a bunch of __END__'s and if(0)'s around to find the failing testcase.
• The major benefit of test suites, IME, is the repeatability. I can make a change, then run 5000 tests to make sure nothing else got bolloxed up. That's nice.
• You're a maintenance dev. I didn't write a lot of .t tests as a maintenance dev. Frankly, it wasn't cost-effective.

---

My criteria for good software:

1. Does it work?
2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

I've done it a few times. It takes a lot of self-discipline but it is very rewarding. Besides having the tests going forward to maintain code health, the mere act of writing tests tends to expose use cases, UI, and design issues you weren't considering or were coming from bad assumptions. It appears to take more time up front but the time it saves in focusing you on "fixing" the tests with new code and not refactoring things you finished as a half-baked prototype can be immense.

I've had the best luck with writing the synopsis first—the stuff that goes near the top of the POD. This way I get a simple use case from start to finish. Usually that shakes out the knees and elbows of the interface I had in my mind. I can streamline things before I've coded any decisions. Once that looks good, it's pretty easy to translate into test cases.

I've had the best luck with writing the synopsis first—the stuff that goes near the top of the POD.

Yes, I was going to say something like that. I don't always write tests first (writing them soon is good enough), but writing (some of) the documentation first is almost always helpful. I find it encourages me to simplify the interface: I get rid of unnecessary options because they're too much of a pain to write about.

But it sounds like the OP has not yet gotten in the habit of putting almost all of the code into modules, and that's really the first step toward sanity. For one thing, it makes it a lot easier to write fine-grained unit tests.

I'm much more likely to write the tests first when modifying existing code, so as to verify behaviour before and after the change.

I used to hate tests, but now I love them.

When possible, I prefer to write the tests first - and yes, it can help you catch a kludgy implementation earlier.

Sometimes, when refactoring older code, I find it easier to do the rewrite first, because I often don't fully understand what I'm refactoring when I start, so my tests would be way off.

Andy Lester has written some great stuff on the joys of testing. I recommend this (HTML version). If that doesn't convince you of the benefits of a good testing strategy, I don't think anything will :)

It didn't convince me.

The HTML version of the slides, at least, does a great job of outlining testing strategies, policies, and techniques, but it says absolutely nothing about why to test (unless you count "remember: human testing doesn't scale") or what the benefits of a good testing strategy might be.

Me neither.

I'm utterly convinced of both the need for, and the benefits of testing during the development process. My argument is entirely aimed at the methodology and tools being advocated and used.

As for the slides, I got as far as page 12 and 13 and just stopped reading:

1. Redundancy Is Good.

   No! Redundancy is redundant.

   I'll put that another way. It costs time. And time is money. Redundant effort is therefore, wasted money.

2. There are no stupid tests.

   Rubbish! Of course there are stupid tests.

   Testing that perl can load a module and converting the result into a 0.05% change in a statistic is stupid. When left to it's own devices, perl will report a failure to load a module with a concise and accurately informative warning message...and stop. Preventing that stop is pointless. Every subsequent test of that module is, by definition, going to fail.

3. I did go back and scan on, because I felt I ought to. And I could pick out more, but I won't.

When testing tools or suites are more complex and laborious than the code under test; when they prevent or inhibit you from using simple tools (like print/warn/perl -de1), they become a burden rather than a help.

In the car industry (in which I grew up), there are many ways of testing (for example) the diameter of a hole in a piece of sheet metal. You might use a lasar micrometer. You might a set of inner calipers and a set of slip gauges.

The favoured test tool? A simple, tapered shaft with a handle and clearly marked, upper and lower bounds. You simply poke it in the hole and if it passes the lower bound and stops before the upper, the hole is within tolorance. It is simple to operate, Is extremely robust in use. And takes no time at all to use. It's easy to use, so it gets used.

The Occam's Razor of test tools. I seem to think that Andy has an appreciation for OZ. He should consider applying it in this case also.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

"Human testing doesn't scale"? Maybe I should have linked to another of his slide presentations that has more background info than practical code.

But for me, from experience, leaving smoke running hourly as a cronjob (that emails me if there's errors) when I'm developing has saved me a pile of debugging. Quite a few times, I've assumed something, only to find that that assumption broke another assumption elsewhere in the code.

I know I'm far too lazy to manually test every time I add new functionality or optimizations, so knowing that I'm going to get an email if there's either something wrong with my code - or that a test is wrong because functionality has changed - is invaluable.

But, if it's just you on the code and it's never going to grow to become a monster(say 10,000 lines or more), then you might not reap the full benefits, or think it's worthwhile. On the other hand, you might be pleasantly surprised at how much it can help :)

XP is big on this. Not that I've ever managed it myself ... 8-)

http://www.extremeprogramming.org/rules.html

For me I guess it depends on the project, the timelines, the users, the requirements. Usually the users don't know what they want, so it is a constant iterative process of prototypes until they say "that's it!" and when you say you want to start again doing it properly it usually goes quiet ... and the prototype goes into production.

I acknowledge that BrowserUK has a point on the numbers, dots and percentages scroll, but I disagree in that this should prevent one from writing tests first. I usually write one (or more) test file(s) per sub, which again makes it fairly simple to narrow down on specific code, and using diag and the more verbose functions of Test::More et al. should render it obvious what went wrong in the event of failure.

As for pitfalls, I think of easy of change. Having both a test suite and code (as opposed to just code) means there is more to change. For smaller scripts where you feel very confident, having to rewrite the tests and the code can be burdensome. Then again, that may be a good thing as it forces you to think while letting you change one section at the time and making sure that it all still works as intended. In overall I don't this this should prevent you from writing (at least some or most of the) tests first.

What helped is to move the functionality to modules, because IMHO modules are much neater to test than scripts.

So I started to write modules that do 95% of the work, and the scripts that use these modules just do some command line processing, but no real logic.

And yes, often I write the tests first. That's especially useful if you have a small, complex piece of code. Like a recursive regex that parses nested constructs.

Then you can build a basic regex, run the tests, refine it, run the tests, ... until you're done. And you don't fall into the trap of modifying something and getting what you want, but having undetected side effects. (At least you're not trapped that easily ;)

What helped is to move the functionality to modules, because IMHO modules are much neater to test than scripts.

True, but it doesn't have to be like that.

Keep the script, but encapsulate all code in subroutines except the last line below. The last line ensures that one can require the script without actually running its' contents. I argue that this script is no harder to test than a module.

# script.pl
sub foo { ...; $retval; }
sub bar { ...; $retval; }

sub main {
  foo() or die();
  bar() or die();
}

# Call main only if no stack.  Non-empty stack means we're testing.
# See chapter 9 of Perl Testing - A Developer's Notebook
main() unless caller;
[download]

--
Andreas

I always write tests first when writing a library, especially one to be used by others. A while ago I summarized the benefits of doing that in: Re: Winning people over to better development practises (TDD).

The trick is to truly design it, completely, first. Yes, all of it. (Oh yes, you can!)

As we all know, the hardest thing about computer programming, if there is a “hard part,” is deciding what to do; not the fairly-rote procedure of actually doing it. But people drop into this self-taught mentality of “just banging stuff out,” just like they first did when they were mastering the language. And then they get to be managers, or worse, and they're still doing it all the same way – they're “just banging stuff out.” Without management-skill (which generally has not too much to do with programming-skill), projects are always missing deadlines and expectations, which those people mistake for “gee, programming sure is hard work.” They vindicate their own actions to themselves, because it's always been this way (for them...), and they never realize that there could be a better way. So, some very smart, very sincere, very well-intentioned people can go an entire career and never bother to learn how the process works. (They think they know.)

You've heard it:   “Don't just plan there!   Do something!”   :-/

When a project gets big or important, it's no longer adequate for “one person” to be making all of the decisions. Not only is this impractical from a timing standpoint, but the true complexity of the project reaches a point where a single individual is not likely to possess all of the essential experience. You need to have several active brains on the project, with eyes wide open and individual responsibilities, and with the sanction to disagree with one another... and with you. To make a project of any size really work, you must borrow from the greater field of engineering discipline and formalize the task of project management, as well as the programming, the documenting, the testing and all the other “technical” things. You can be responsible for a project, and be very pivotal in making it succeed, and not write a single line of the source-code yourself.

O'Reilly has recently published a definitive book (which doesn't have an animal on the cover...) called The Art of Project Management, by Scott Berkun, (ISBN-13: 978-0-596-00786-7) which should be required reading. In a similar vein is Debugging the Development Process, published by Microsoft Press. (It's part of their “Software Engineering Classics” boxed-set.)

I like to remind people that if you wanted to get a new addition built onto your house, and the first day the guy showed up with all his workmen and they started cutting boards and banging nails while the foreman went inside to ask you what you wanted ... if as the days went on they built something and tore it down and built something else, obviously “just making it up as they went along” ... why, you'd throw that guy out of what's left of your house and call your lawyer! So why this sort of nonsense is deemed to be de rigueur in a software project is quite beyond me. But it happens every day, and millions of dollars get blown that way.

You can do these things on a smaller scale too; make it part of your overall daily practice. First of all, you need to plan ahead so that you're not bumping into “crisis mode” and throwing every concern out-the-window. (Never agree to a “schedule” that you don't agree with. Never work to a “wiki” as a spec.) Instead of building a great-big thing only to discover that it doesn't work and have to scrap it, probe the entire project from a higher-level before you start ... looking everywhere the waters seem to be disturbed to explore what kind of rocks might be underneath that particular area. In the case of Perl, use CPAN (and this forum!) very aggressively so that you can take advantage of someone else's experiences (ka-whack! ouch!!) rather than repeating them.

Microsoft Project® can be your greatest ally:   you can “work it out on paper!” If you can't say day-for-day what you're going to be doing and what milestones you will reach, you're not ready to start work. If you find that you're not hitting your deadlines squarely, you need to stop right away and fix your schedule:   the schedule reflects the plan, so if the schedule's non-functional, so must be your plan... and the schedule is simply your bellwether; your canary.

I like to remind people that if you wanted to get a new addition built onto your house, and the first day the guy showed up with all his workmen and they started cutting boards and banging nails while the foreman went inside to ask you what you wanted ... if as the days went on they built something and tore it down and built something else, obviously “just making it up as they went along” ... why, you'd throw that guy out of what's left of your house and call your lawyer! So why this sort of nonsense is deemed to be de rigueur in a software project is quite beyond me.

In software terms, we would call those workmen the compiler. I leave fixing your example to match the reality of both construction and software to the reader.

The trick is to truly design it, completely, first. Yes, all of it. (Oh yes, you can!)

Has that ever worked for you on a project that took longer than a week to complete? Yeah, me neither.

(The compiler...)

Those workmen have nothing to do with “the compiler.” To continue your peculiar analogy, the compiler would roughly correspond to a cement-mixer:   a necessary, but passive, tool for the job that is used by every single job of its kind.

---

(Have you ever...)

For a project consisting of more than a quarter-million lines of (not Perl...) source code, lasting nearly ten years ... yes, it did.

---

Even though the thought of meticulous planning somehow earns responses of “why bother” or “software is different,” I'll stick to my experienced guns.

Just as you can design a physical machine, or a building, or a golf course, or a shopping center “on paper” in advance of building it, and even accommodate fairly on-the-fly changes to those plans “on paper,” you can design software systems in advance, and keep those paper-plans up to date.

When people spend $30,000 or so on an addition to their houses, they take for granted this sort of discipline. Yet when they spent $3,000,000 (maybe without quite realizing that they're doing it, or that they're wasting more than half of it) on a software-project, they get lured into thinking that somehow the rules have changed. No, they haven't:  you're still paying someone to do something, and you still expect to have reason for confidence that you'll actually get it.

When you are “in the trenches” on a project, as most “Perl programmers” are, you just don't see the financial burn-rate. You don't think of your salary as an expense. It might never occur to you that the company is spending a million a year on what you're doing, but that's actually just a small project:   it could be much more. Software projects are actually very-big capital investments that traditionally do not get the same scrutiny and treatment that other smaller investments routinely receive. Don't ask me why...

So what does all this budget-talk have to do with testing? Everything! Physical construction projects look simple as you whiz past them at 65mph on the freeway, but they're meticulous undertakings. That's why you whiz across thousands of bridges in the course of your travels and (almost...) never fall into the river. Software has never had that sort of track-record, and here's part of the reason why.

Sundial, I like your writing, and he who writes well, demonstrably thinks well.

And I appreciate a proper requirements cycle, but I must concur with chromatic here. Design first with all its attendant paraphernalia (artifacts which often obscure the fact that implementation is design) is often too brittle.

As the sages of the middle way opine, you are, in this post at least, heading too far to one side of the game.

give me one or two scribblers who can understand business needs, design and implementation, and I'll be in Scotland long before the project managed, designed, and handed off to in shore/offshore coders group has found out how little the design had to do with meeting the requirements

I'ved tried the design first way. with ten pms to two developers. I was the implementer. We could have saved so many cycles if we had fired 9 pms, and had me collect the requirements. In that case. ymmv.

Furthermore, a wiki is a tool like any other. wikis don't kill projects. people kill projects.

update: design prayerful-gnalia. where is the qualia?

it [testing] cannot be used as a substitute for hard thinking.

Of course not. On the contrary, testing and thinking go hand in hand. Personally, I think writing tests forces me to think more thoroughly than I otherwise would have.
--
Andreas

On the other hand, you'll get serious XP on Perlmonks for saying you test first.

EDIT: Thank you, downvoters, for making my point.

Before learning Perl I wrote some software in Eiffel, and they had that "Design by Contract" paradigm very early, and I used it extensively.

It helped, but it didn't replaced tests.

Why? Because the code has to be run in order to check these precoditions, postconditions, class invariants etc.

And if a method isn't called during testing, all your assertions are vain.

I ended up rerunning the application all over just to make sure these assertions were executed and checked. But the test suite should have done that, not me manually.

There's nothing wrong with DBC (and I like the fact that Perl 6 will come with it by default), but it's just not enough.

And if the code (or "method" -- I need to get my OOOrthodoxy checked) isn't called when you use the program, it probably doesn't matter. If you're writing code you don't use, you've got a more serious problem.

"Test first development" seems to me to be a blunt tool for people who don't understand their code, or even make a serious effort to do so.

Thank you, downvoters, for making my point.

Martyr much? I didn't downvote you, but I do object to your implication that I don't understand my code or make a serious effort to do so.

Feel free not to use any of my code, if you think I'm such an amateur.

I was speaking more about the design side, using it more, to borrow your words, as a blunt tool to design the project. My idea (which as I guessed is not even close to a new concept) was just to combine the time spent spec'ing out the project with the time that would be spent later on writing tests.

What do you do as your first step in the design?

I think about the problem and solution, trying to come up with its governing laws. Sometimes I write math and pseudocode in a notebook.

Why don't I think of the big picture? My user or my manager is usually calling (again) asking why the task isn't finished yet. ("I asked for something that was pretty straight-forward; why are you worrying about bells and whistles I may never need?". Until, of course, it is needed...)

No, I don't. I have tried it a few times, but it's not appealing to me. Basically, what you attempt to do with the "write tests first" paradigm is delegate the most important checks to the machine: that the code you have written is correct.

Now, correctness is an elusive word to define, because correctness of a program depends on its requirements, and requirements are seldom, if ever, fully consistent or even complete. Nevertheless, the point is that by writing tests you describe the procedure how to check that the code you have is correct, and then you can automate the correctness checks by making the machine run them.

The benefits of it are that if you can fully formalize what correct means, you can check the program time and again -- or rather you can just run the test framework and the machine will report its findings. Sounds like good application of laziness.

However, it only works if you can formalize -- if you can write comprehensive tests in the first place. There are several reasons why it's difficult or even impossible.

First, there's the question of what input to use. In all but trivial cases (i.e. no input, or a constant expression), the input space is too large to be comprehensively searched. So you settle for "corner cases" and usual sources of error. Ignoring how you can come up with these (maybe you develop skill as you write tests), how do you know you have it all nailed down? How do you know you are testing with the right input, or that you have included all corner cases? It seems to me that if you knew this, you wouldn't need to test; you could simply prove (by deductive or inductive reasoning) that the piece of source code always works.

Second, the programming language you use for tests is the same as the programming language you use in programming the application. This is natural, since you want easy integration, and you want executable test specifications. However, this is likely not the best choice. I know I can't express certain things in Perl that I would need in writing tests, or more specifically formalizing how to check correctness. I don't mean fancy things such as pi calculus or some nonsense; I mean, for example, being able to write a quantified expression without having to put it in the form of an implicit or explicit loop.(*)

Thirdly, the usual benefit cited is that writing tests forces you to think about the interface. Thinking about the interface and sketching different alternatives is actually a very good thing, though why it has to be disguised as testing eludes me.

What I normally do is start writing documentation. I write the synopsis, some or most of the description, and then start documenting the non-existent public interface. For each function or method I think would be useful (and for all functions and methods in general), I write a line or two of example code how to call it, describe what kinds of input it expects and accepts, and what kind of output it will give (or how it will change the state of the object, say), and what kind of error conditions to expect.

Sounds ridiculous? Try it. What you simply have to do during writing is actively think what you even want from the module or piece of code, how you should build the interface, and most importantly, how to distribute functionality among the modules. Often I notice that I'm missing an abstraction or a class, and I repeat the same steps for that. It's not the silver bullet, of course. It doesn't always work.

The usual argument against this is that then you have to keep the documentation in sync with the code, or that they don't really match, or that you have to do a whole lot of extra work. The function of the documentation is not to replace the source code, or to describe how the program works, as you probably already know. It's to supplement the missing pieces that you lose when you encode your ideas in a programming languages: The whys. The intent and purpose.

Coming back to testing, the fundamental difficulties in instructing the machine how to check the validity and correctness of source code are that 1) you cannot encode everything in executable source code, and 2) the dumb machine will only check the things you have explicitly told it to. Since it's clear that you don't know exactly what you're doing if you have to write tests, it's wishful thinking to expect the machine to report errors in places where you haven't thought were possible.

One important aspect of testing I agree with: checking for implicit assumptions in the run-time environment, for example when installing the piece of code on an entirely different kind of platform than where it was developed. But this is only because those assumptions are not documented anywhere, and even if they were, there would be too many to take into account.

(*): "For all x, P(x)." Here P(x) is some Boolean statement about x. If I'm writing tests, I don't want to loop over the entire input space and check P(x) for all x. I would just want to write the condition down and make the machine deduce for me if it applies or not. But if we had that, we wouldn't need human programmers in the first place.

In fact, I have code which is being used in production, just a couple of weeks into development, and the scripts which are actually being run are in my t/ directory. I've run a module in development for weeks at a time like this, from the t/ directory, doing production work, as I continue to develop the module, adding features, monitoring output, refactoring early efforts, tightening up pieces I have to handle manually until its ready for prime time.

I find it very useful and helpful to watch a test fail, and then to resolve each error in turn as it presents itself, until the test passes.

I code in vim, using konsole on a gui desktop to access a command line. Konsole permits multiple tabs, each in its own directory, with its own tab label, perhaps even connected to its own server. I'll use three adjancent tabs all in my sandbox/My-New-Project/ folder. One I use to vim lib/My/New/Project.pm. The next tab invokes vim t/20-my-new-project.t. The third is used to invoke `time perl t/20-my-new-project.t`.

If a database is involved, a fourth tab will give me a psql or mysql prompt to the affected database, even when my test scripts are directly monitoring the results in the database, as well. If I'm writing a cgi script, I'm likely to also have a browser opened to localhost/t/testscript-my-new-project.cgi, as well.

I don't write tests first, so much as along with. Usually just a test or three at a time, then the code it tests, then a couple more tests, and some additional code. By the time I've built a ->new() method there may be a dozen, or two or more tests making sure that all the sanity checks are working.

I've found that I spend far less time debugging this way than I use to. No more chasing errors around. When I have three or five tests which reasonably exercise an internal method, changes are less scary.

I also use Devel::Cover, is it, to examine the coverage offered by my test suite. When I get stuck or bored moving forward in implementing a design, I might spend some time looking at how I can improve coverage of my test suite.

My first supposedly Test-Driven-Development project got me working code, and a well populated t/ directory. But I was aghast at how few branches were actually exercised by my tests when months later I applied Devel::Cover to it. Now I run my test suite under Devel::Cover from time to time as I go and fill in the holes. It leaves me far more confident in the results I produce.

And as one client mentioned to me, it left her far more confident in my work as well. I sent her the test results. The well named tests (after bugs or variations in the data) left her confident not only that the module I built for her would appropriately process her test data she had sent me, but would also handle the 5,000 records that module was written to process, as well.

-- Hugh