The testing survey results are fascinating and, in some ways, discouraging. There is, as always, a bias in online surveys (you have to both know about the survey and have a desire to take it). For example, fully 66% of respondents claimed that their primary code base has a test suite but it's been my experience that the number is far smaller.

I originally posted this to my LiveJournal account, but brian_d_foy offered to put this on Polldaddy under his account. He also expanded a number of areas in the poll and generally made it more useful. Thanks brian.

Also, I would love to expand this survey and post it to my O'Reilly blog to get a better cross-section of programmers, but most of the free survey software out there is limited in the number of responses and/or features. Anyone have any suggestions?

Survey Title: About your test suite...

Page 1

Q.1 Does your primary code base have a test suite?
Answer Count
No 56 (34%)
Yes 111 (66%)
People who answered question: 167 (100%)
People who skipped question: 0 (%)

Page 2

This page was only answered by people who don't have test suites

Q.2 Why don't you have a test suite?
Answer Count
**other** 5 (5%)
Don't have time 26 (26%)
Legacy code 25 (25%)
New Option... 2 (2%)
Don't know how to test 15 (15%)
Too hard to test our code 14 (14%)
Management doesn't approve 12 (12%)
People who answered question: 56 (34%)
People who skipped question: 111 (66%)

Of the five people who answered 'other' to why they don't have a test suite, their reasons were as follows:

  1. Research - exploratory only
  2. Cross-compiling to platform with no testing tools
  3. still learning how to write them
  4. New job where I'm pushing for testing, revision control, etc
  5. mod_perl

Q.3 Do you plan on adding a test suite?
Answer Count
Yes 35 (63%)
No 21 (38%)
People who answered question: 56 (34%)
People who skipped question: 111 (66%)

Q.4 How soon do you plan to add a test suite?
Answer Count
**other** 6 (16%)
Within a year 16 (42%)
Within six months 15 (39%)
This month 1 (3%)
Next Month 0 (0%)
People who answered question: 38 (23%)
People who skipped question: 129 (77%)

Of the six people who answered "other" to how soon they plan to add a test suite:

  1. When hell freezes over or "there are no more features that could be implemented any more"
  2. someday
  3. as soon as we get round tuits
  4. Never
  5. as I have time
  6. Starting now incrementally

Q.5 What are your test suite plans?
People who answered question: 27 (16%)
People who skipped question: 140 (84%)

Of the 27 people who answered this question:

  1. learn how to make a test suite...
  2. Start with basic testing until I understand it better. Then move up to more complex suites.
  3. start from the ground up. Looking at what is a good way to start.
  4. Unfortunately planning and writing test take time and effort. When documentation consists of your ability to read badly written legacy code and management has a incredibly urgent/ninja rush button firmly pressed down on every change request (no matter how small) it is hard to introduce a culture of testing.
  5. None yet.
  6. Start with some of the more easily separated, more sensitive areas of functionality, and build from there.
  7. To use a test suite and learn how to use one.
  8. vague
  9. None
  10. Depends on the organization of the final code. Currently it is a mix of Perl and Java. Once the research phase firms up and we establish a clear direction, we will use a test harness as appropriate for the language.
  11. I'm still learning test based development, I've read a few articles and tutorials, but just started a project that will have test suite from the beginning.
  12. Put tests into my next project
  13. none yet
  14. Refactor code to make it testable. It's a website whose html is generated by print statements and subs return strings of html. So: subs -> data html -> templates
  15. None
  16. If the tools were available, I'd at least start testing my models as soon as we have an initial version of our current programs out.
  17. add tests to existing code as I can; build new code with tests in place.
  18. From-scratch rewrite of large parts is coming up; will have tests. Tests for legacy codebase not considered worthwhile despite regular "oopsies" when rolling out changes.
  19. Haven't given it a ton of thought yet. We have a mixed code base of ASP.NET (VB.NET/C#), PHP, Perl, static HTML, JavaScript, etc and I'd like to either find something that can work between all of these or I need to take the time to get more familiar with various testing frameworks.
  20. write testcases for running code and collect the TAP output for statistics. We need test in the runtime environment, at compiletime there is not much to test (ie: no db, and faking one is no guarantee)
  21. Ridiculously, management pays some junior person to spot-check whatever they feel like checking at the current moment. They think this is an adequate testing policy.
  22. slowly add unit tests as code is touched.
  23. test for every fu**** function
  24. Require tests of all new code. Start with some simple back-fitting of tests. Not sure how to sanely test the large DB queries we do. Not sure how to test the rendering in the front-end.
  25. Start with (a) parts that are used most prevalently (b) parts that are self-contained Write tests for these and gradually work outwards. Adding tests for all new code.
  26. After a code refactoring to allow for more testable chunks, will add in tests.
  27. TAP

Q.6 What would help you get started?
People who answered question: 37 (22%)
People who skipped question: 130 (78%)

Of the 37 people who answered this question:

  1. teach me how to make a test suite
  2. new project
  3. Time
  4. The boss oking it. Several easily accessible tutorials with examples. Good papers on why testing is needed.
  5. Do I use Test::More, Test::Harness, etc? I have lots of stuff to look at but I haven't sat down and figured out where to start, at least what is the current, suggested/approved way to start testing.
  6. Tutorial on options and configuration.
  7. Test::Simple where you can easily write tests in a manner similar to the expect programming language without needing hooks. Most test suites seem aimed at unit tests. Most of the shoddy code I deal with lives outside of modules (usually in web pages).
  8. A formal block of general instruction and guidelines.
  9. Convincing people of that it's worth the time spent.
  10. More time to study.
  11. a framework for testing CMS's af web-applications
  12. Training and more time to devote to testing.
  13. More programmers. :-)
  14. Having more expertise, but I know I can't achieve without actually starting to use them.
  15. A good tutorial.
  16. more time
  17. time from management to work on it
  18. 1. A tutorial on how to test. 2. A "Test Patterns" like "Design Patterns"
  19. More approved time by management
  20. A complete rewrite of our codebase (testing is nearly impossible as is), and management that doesn't consider automated testing to be a complete waste of time.
  21. Someone to come in with a Clue Bat and smack the entire chain of management around until they submit.
  22. Tools for iPhone testing!
  23. - The available book that everyone cites (Perl Tester's Notebook) doesn't work for me as a 'getting started' manual, so having alternate resources would be a great start. Maybe I just don't know where to look. - Time to work on it (this is a corporate issue). - Actual hands-on help from someone who knows what they're doing. I get so stuck sometimes, I dont' even know what to google. - A "testing dojo" at the local PM meeting...maybe I'll suggest that. :)
  24. Having a boss that understands the benefit and cost reductions of implementing automated testing
  25. Hastening the planned feature freeze of the legacy codebase...
  26. Info on popular cheap/free testing frameworks for those languages/frameworks.
  27. documentation on testing strategies. Most talk about ok(4+4,8) but that is not really a testcase IMHO, we also test response times as this is critical to user experience. We could just log this in the db, but I'd like to read about testing strategies for this using TAP (ie: the actual response time is irrelevant, it's either 'good' or 'bad' which can be determined fuzzy wise)
  28. New management. :)
  29. time
  30. time
  31. More time.
  32. Tutorials with non-trivial examples.
  33. How-Tos that show best practices for web site testing - preferably with Catalyst.
  34. A better build system
  35. Understanding testable code.
  36. A raise and a bonus.
  37. A down-to-earth howto.

Page 3

This page and beyond was only answered by people who do have test suites

Q.7 What test framework the test suite is based on?
Answer Count
**other** 25 (17%)
JUnit 11 (8%)
Test::Harness 106 (74%)
XUnit 1 (1%)
ObjcUnit 0 (0%)
OCUnit 0 (0%)
People who answered question: 143 (86%)
People who skipped question: 24 (14%)

Of the 25 people who answered "other" to the "What test framework" question:

  1. Is what I plan to start with
  2. None
  3. Test::Class
  4. Undecided
  5. home-grown c inspired by Test::More
  6. DejaGNU
  7. N/A
  8. N/A
  9. homegrown
  10. home spun
  11. Test::More
  12. schemeunit
  13. N/A
  14. Test::More
  15. Test::More
  16. Test::Class
  17. Test::More
  20. inhouse
  21. Custom. Much simpler and easier than xUnit
  22. Internal
  23. Test::More
  24. NUnit
  25. Mine

Q.8 What is the primary language the test suite is in?
Answer Count
Java 8 (5%)
C / C++ 8 (5%)
**other** 8 (5%)
Python 3 (2%)
Perl 124 (82%)
Ruby 0 (0%)
People who answered question: 151 (90%)
People who skipped question: 16 (10%)

Of the 8 people who answered "other" to the "primary language" question:

  1. Would be perl, but you have no skip ahead option in the poll
  2. N/A
  3. Undecided
  4. scheme
  5. N/A
  6. JavaScript
  7. C#
  8. Scheme

Page 4

Q.9 How many tests are in the largest test suite you work with?
Answer Count
Fewer than 50 tests 32 (22%)
50 to 100 tests 19 (13%)
101 to 500 tests 35 (24%)
501 to 2,000 tests 25 (17%)
2,001 to 10,000 tests 16 (11%)
More than 10,000 tests 16 (11%)
People who answered question: 143 (86%)
People who skipped question: 24 (14%)

Q.10 How long does your test suite take to run?
Answer Count
Fewer than 30 seconds 31 (22%)
30 seconds to 1 minute 23 (17%)
1 minute to 5 minutes 36 (26%)
5 minutes to 10 minutes 9 (7%)
10 minutes to 20 minutes 13 (9%)
20 minutes or more 26 (19%)
People who answered question: 138 (83%)
People who skipped question: 29 (17%)

Q.11 What percentage of your code is covered (rounding down)?
Answer Count
Don't know, but I do coverage testing 6 (4%)
Don't do coverage testing 47 (34%)
**other** 3 (2%)
Less than 50% 21 (15%)
50% to 74% 18 (13%)
75% to 84% 16 (11%)
85-94% 12 (9%)
95-99% 15 (11%)
100% 2 (1%)
People who answered question: 140 (84%)
People who skipped question: 27 (16%)

Of the three people who answered "other":

  1. What tests?
  2. less than 1%
  3. A tiny fraction, so far

Q.12 Do all tests pass?
Answer Count
No 28 (20%)
Yes 110 (80%)
People who answered question: 138 (83%)
People who skipped question: 29 (17%)

Some of the results I found particularly interesting are as follows (I present hard numbers, but it's worth remembering that they are all in relation to those who took the survey):