Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re^3: too much testing?

by leriksen (Curate)
on Oct 03, 2005 at 01:25 UTC ( [id://496796] : note . print w/replies, xml ) Need Help??

in reply to Re^2: too much testing?
in thread too much testing?

A comon pattern I find in my testing code is

my @dieings; my @warnings; my $rc; eval { local $SIG{__WARN__} = sub {push @warnings, @_}; local $SIG{__DIE__ } = sub {push @dieings , @_}; $rc = $obj->method(@params); } is(@warnings, $expectedWarningCount, 'correct number of warnings raise +d'); like(shift(@warnings), qr/expected pattern of first warning/, 'first w +arning message correct'); like(shift(@warnings), qr/expected pattern of second warning/, 'second + warning message correct'); ... is(@dieings, $expectedDeathCount, 'correct number of deaths raised'); like(shift(@dieings), qr/expected pattern of first death/, 'first deat +h message correct'); ... is($rc, $expectReturnValue, 'method returned required value'); # or, if method returns an object isa_ok($rc, 'Expect::Class', 'received expected class'); ...

...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

Replies are listed 'Best First'.
Re^4: too much testing?
by geektron (Curate) on Oct 03, 2005 at 04:14 UTC
    that's an interesting way of handling it.

    funny thing is, people seem to be gravitating to the part about catching $SIG{DIE} and not even noticing the title, or part of the real "meditation" ... though i guess not explicitly commenting on it is a commentary in itself.

    being the novice tester, i really am wondering when 'enough is enough' ...

      OK, so as to your question(s) - are you testing "too much"

      I have strong opinions on this, based on years of writing code that has experienced jut about every failure mode imaginable.

      I have a personal belief that you have tested enough when you feel you have exercised as much of the code as you reasonably can. I use Devel::Cover to tell me how much I haven't done, and use my experience and judgement to make informed decisions as to what tests I will add for the remains.

      For example, do you need to test what your code does if read() fails ? If so, writing test cases to do this is a bit difficult, but if you need it, do it.
      Otherwise, code and test on the assumption "read() will never fail" and move on.

      Devel::Cover isn't perfect, and sometimes it can be very difficult to work with - for example, it complains about my $class = ref($proto)||$proto; as having an untested path in most "reasonable" usages. You have to write a test that calls the constructor in a manner you would expect to see for a function-based module e.g. Module::new(), not Module->new()
      So you need to ask yourself "is testing for people calling my OO methods as normal functions something I want to do ?"

      If it is important to you, you help newbies who do "Module::new" by mistake.

      If not, ignore the report of the untested path from D::C and move on.

      As an aside, I wrote a meditation on what I had to go through to get a reasonably large body of modules to get to 100% coverage as reported by D::C - see Lessons learned from getting to 100% with Devel::Cover about my struggles and discoveries from doing a deep testing effort.

      ...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

        i do remember reading that meditation, and i'm going to re-read it in a few minutes.

        Devel::Cover is going to be my basis for "percentages". in this particular case, trying to achieve 100% coverage may not be practical at all, at least not pre-application release. (i say this *only* because these are single-site modules and not a public-consumption distro. i'm sure i'll add more features and tests later, and can try to come closer to 100% coverage.)

        in any case, thanks for some opinions on knowing when to say "enough is enough".