Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re^3: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)

by dragonchild (Archbishop)
on Oct 24, 2005 at 01:10 UTC ( [id://502355]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
in thread Neither system testing nor user acceptance testing is the repeat of unit testing (OT)

Oooh. This is where I start to get a little antsy. In my extremely un-humble opinion, white-box testing is a CodeSmell. If you need to look at the code instead of the interface to determine what tests you need to run, your interface isn't correctly designed. Correct interface design and mocking will allow you to black-box all your unit-tests.

That's a big statement, but I'm willing to be proven wrong. (Yes, that's a challenge.)


My criteria for good software:
  1. Does it work?
  2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Replies are listed 'Best First'.
Re^4: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
by leriksen (Curate) on Oct 24, 2005 at 01:53 UTC
    OK - in reponse to your challenge, I've got one word to say to you - Exception Flows ;-)

    Say you have a function/method, as part of your interface, that writes a file out somewhere. This isnt the main role of the function/method, its just part of the work it does for you. The normal flow is to open the file, write whatever, and close the file.

    But there is also code to deal with being unable to open the file, not all the data being written or the file not being able to be closed.

    How these failures get communicated back to the client can vary too - from not saying a word through to throwing an exception or even calling exit().

    My opinion is that the developer is required to setup test cases for each of these failure modes, and this requires a white box approach. Knowing that the code writes a file may not be inherently knowable from the interface (whether it should be is a different point), but only by looking at the code do you _know_ that it writes a file, and that it does different things for different failures. Even if the interface doco states that it writes a file, detailing the different failure paths is almost certainly not part of the interface design or doco (but may well be part of the interface _implementation_ doco).

    The function/method writer can test all the normal stuff just by reference to the interface, but setting up file-open failures and checking that the function does the right thing needs more detailed knowledge related directly to the code. This is where tools like Devel::Cover can help - exception flows pretty much stick out straight away as bits of code, related directly to your chosen implementation of the interface, that you have got to write test cases for.

    ...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

      Ouch. Any possible exception that might be thrown by you needs to be listed in your interface, because it's a possible "return value". In addition, you also need to list every single interface that you use, because your caller might need to look up their interfaces to see which exceptions they might throw.

      To recap, exceptions most definitely are part of the interface spec.


      My criteria for good software:
      1. Does it work?
      2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
        I'm not limiting myself to exceptions and throws - I'm talking about exception flows - parts of the cod that gets executed when something goes wrong. This may result in an exception getting thrown, but only if the designer (probably not me) says I can throw - should the design be at that level of polish. Maybe the designer says - thou will not throw an exception in your implementation - I might disagree, but thats what I _have_ to do. I dont think I'd have a job long if I said "screw you , I'm throwing one anyway."

        What I am referring to are testing the decision I made for the implementation of the design. _If_ I chose to use an on-disk file as part of my solution, and _if_ I wrote code to handle different I/O failures, I would want to make sure I wrote unit test cases to check I correctly implemented handling the different failures. Maybe I just handle them and dont tell the caller anything, maybe I log it and continue on. Whatever I chose to do, I should test I implemented correctly. How wise my on-disk solution is, that's a different question.

        In short, those things I can test from the interface/design point-of-view, I agree with you completely and I should test black-box, those things that are artifacts of my implementation, but not part of the interface/design, I should test black-box if I can, and white box where I need to. Its all about making sure you tested "everything" - well it is for me, unit testing my code to as close to 100% coverage as possible is what I aim for. In the OP's case, calling a bad SQL wasnt something their testing had caught. Somewhere, if they ran a coverage report, they might see where that SQL was being injected, as a piece of code that hadn't been tested - a very white box approach.

        ...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

Re^4: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
by tilly (Archbishop) on Oct 24, 2005 at 06:35 UTC
    Since you've thrown down the gauntlet, let me pick it up.

    You present a false dilemma, suggesting that you only get to do one kind of test. And then you present an argument that black box unit tests should be more important.

    But the fact is that you can do both. And they pick up different kinds of errors. I fully agree that all of the tests of the basic interface should be motivated by the interface specification, not the implementation. That is, they should be black. But corner cases and special cases in the code are a common source of bugs, and that one may only reliably figure out where those cases actually are by staring at the implementation. (One may guess without peeking, but one only knows after looking.) So including white box unit tests to smoke out possible bugs in corner cases catches errors that black box testing may not.

    Therefore after you write your black box unit tests, there is value in adding white box tests as well. Given a positive return from doing the work, one should do it.

    This is also why white-box measurements such as coverage percentage (see Devel::Cover) have value.

    UPDATE: minor punctuation and added "white-box" to the last sentence to clarify my point.

      This is so funny. I feel that your argument against my point only ends up supporting it. "corner cases and special cases in the code" are very much a common source of bugs. However, I think that having corner/special cases is a CodeSmell. (Yes, I'm being idealistic here - there is no application whose code smells like roses.)

      Plus, imho, Devel::Cover is a measurement of the success of your black-box tests vis-a-vis your interface. If you have lines of code you don't have tests for, that's a problem that needs to be addressed. Granted, you might address is by looking at the code to see which interface components either haven't been tested or have been over-coded for. But, that doesn't mean the test suite has necesssarily become white-box. I often use coverage to both learn more about my interface and to remove code that shouldn't have been there in the first place. Still black-box.


      My criteria for good software:
      1. Does it work?
      2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

        I think that having corner/special cases is a CodeSmell.

        That depends upon what those corner cases are. Sometimes it is impossible to avoid them because the real world does not behave the way we want it to. A common example would be leap years. You have to have corner cases there. In fact, there are a lot of corner cases in date handling due to how tricky it can be.

        Also, hourly employees generally get time and a half if they work more than a set number of hours per week, but for some states/countries, they can also get time and a half if they work more than a set number hours per day. Those are exceptions to the rule and, as such, may get special treatment in the code.

        Or how about removing data from a database? Many systems merely mark the data as "deleted" buy don't actually delete it so that it can be recovered. However, there may be legal reasons why the data must be deleted (for example, contractual agreements). This may come up as a "special case."

        Many issues which may be a corner case can be developed in such a way that they're not a corner case (deleting database records, for example), but whether or not it's practical to do so depends upon the needs of a given application.

        Cheers,
        Ovid

        New address of my CGI Course.

Re^4: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
by pg (Canon) on Oct 24, 2005 at 02:58 UTC
    "... to determine what tests you need to run, your interface isn't correctly designed."

    There is a big hole in this logic. Testing should not and cannot depend on the correctness of anything else, regardless whether it is the interface design or the actual code. There is simply no dependency, we don't expect it, and we avoid it. The whole purpose of testing is to find faults with the application, from requirements to design, to coding.

    If we assume that the interface design is great, or depend on that, then the entire purpose of the testing is defeated right there.

Re^4: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
by adrianh (Chancellor) on Oct 25, 2005 at 09:29 UTC
    Correct interface design and mocking will allow you to black-box all your unit-tests. That's a big statement, but I'm willing to be proven wrong. (Yes, that's a challenge.)

    I basically agree, with a moderate number of caveats :-)

    • Legacy code. I'm going to carry on adding white-box unit tests to test free code that's not been designed with testing in mind.
    • Code clarity. Sometimes I think the code remains clearer with a couple of private methods than it does by factoring out a separate support class. If I think that I'll quite happily write white box tests if I need to tweak them.
    • Can TDD ever be black box? Since the next test is driven both by the spec and by your knowledge of the implementation you can argue that it's always white box. The kind of unit tests that TDD produce are very different from the sort of unit tests a pure specification based approach might take (see discussion on the TDD list for example).

      I think TDD is only done correctly if unit tests are black box. You write your tests and thereby define your interface, regardless of the implementation behind it. Once you get to the refactoring stage you (may) change your implementation to make your code better organized|more efficient|less redundant|etc. Your refactoring succeeds if you manage to change your implementation without breaking a test. If you then have code that's not being tested, you have one of two cases

      1. Your tests don't accurately describe your interface, because you forgot to test for an outcome that should be part of your spec.
      2. The untested code should be removed because it produces an outcome not included in the spec (because ideally your tests are the most thorough description of the spec)

      In the case of 1. you need to add a test, but you still want to test against the module interface, not against the particular implementation. This is a corner case where you're actually finding a better interface by coincidence while refactoring. If you're writing your tests correctly it shouldn't happen (though IRL it does).

      I do agree that it takes a special sort of mindset to properly separate your test-writing brain from your implementing brain, so that's a practical problem for all but the schizophrenics among us :-).

      In the case of retrofitting tests to code, one might divine the specification from the implementation. I will be doing this for PDF::Template over the next few weeks. However, the tests should still be testing the interface, not the code. I fully expect to find and fix several bugs over the next few weeks by thinking of things this way.

      In the case of private methods ... that's an interesting point. Now, you have a private interface that you're testing against. You write tests against the private interface, then tests against the public interface. Just because you didn't publish the private interface doesn't mean it's not a black-box test. The tests for the private method are against the privately-published interface of the private method.

      TDD tests shouldn't be driven by the implementation. Look at Pugs as a perfect example of this. Half the test-writers can't read Haskell to save their life, but it's still TDD.


      My criteria for good software:
      1. Does it work?
      2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://502355]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others scrutinizing the Monastery: (3)
As of 2024-03-28 15:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found