http://qs321.pair.com?node_id=945069

Sometime in June 2011, I wrote BroadcastWeek.pm, a module to get the broadcast week for a given date.
The input is a datetime object, and if it is not provided, it works with DateTime->now().

In August, I realized that this was a very important module for my projet, and so I wrote tests for it, after asking here - How to test a Moose Role?.

I wrote many tests for all the dates in one week in the current year, and many other dates in the beginning and the end of the the last year, this year and the next three years. All tests passed. I didn't expect any trouble from this module at least for the next three years.

And yet, on December 25th 2011, the last day of the last week of the broadcast year 2011, the module returned the week 12-01 instead of 11-52.

I ran the tests again, and again the tests passed. I checked and saw that one of the tests actually tested for 25th December 2011.

I printed the date in my module where it was failing and it said 2011-12-25. I printed the dates on the test file and it too said 2011-12-25.

So, where was the problem?

Since the value of now() changes all the time, I could only test for hard coded dates. And all my tests checked for a given date. Never for a date and a time. But now() gives me the current date with the time.

I added a test for December 25th, 2011 3:40 PM and ran it, and this time, it failed.

So, I changed the module to convert the date it receives into just the date without the time. And to use today() instead of now().

And so, all tests pass again.

I'm hoping that this post helps someone who is starting with automated testing answer these questions:
1. When should I write tests?
2. What should I test?

Merry Christmas :-)

Replies are listed 'Best First'.
Re: Testing in real life
by cavac (Parson) on Dec 25, 2011 at 17:30 UTC

    I can feel with you. Testing against timestamps and such can be a pain.

    A good friend of mine once wrote an NNTP archiver (searchable HTML). Took him a while to figure out why a specific users messages would always fail to parse. After a week or so of tinkering, this is what he found:

    The message date header was (as far as i can remember) in the form of "Mon 25th Dec. 2011 18:00:00". Except this specific NNTP client did the day of the week calculation wrong (25th Dec 2011 would be a sunday, not a monday). Since this where all some years old messages, finding the problem was not as obvious as glancing at the calendar.

    Turns out that the date parsing module my friend used did the parsing and validation correctly. And it therefore rejected the date given due to having the wrong day of the week. Solution was to remove the day name via regex - and then it all worked out. That's how i learned to appreciate the mantra from RFC1122: "Be liberal in what you accept, and conservative in what you send"

    As for calendar week (week of the year) which you seem to use: Yeah, like all Date and Time calculation, this has been historically quite confusing. And given all this leap year stuff, it's hell to get right all the time (excuse the pun). You're quite fortunate that you don't need "seconds since ..." resolution, that would be impossible to get right by calculation: leap second "For example, currently it is not possible to correctly compute the elapsed interval between two instants of UTC without consulting manually updated and maintained tables of when leap seconds have occurred. Moreover, it is not possible even in theory to compute such time intervals for instants more than about six months in the future."

    So, in effect, i can feel your pain. And i thank you for this meditation.

    BREW /very/strong/coffee HTTP/1.1
    Host: goodmorning.example.com
    
    418 I'm a teapot

      Thanks cavac. It's nice to know that this is the first post you ever approved.

      The mantra from RFC1122 is also very interesting and should help me write better programs.

Re: Testing in real life
by CountZero (Bishop) on Dec 26, 2011 at 03:40 UTC
    Another mantra you might keep in mind for any reasonably complicated enough program or module is "when your tests do not fail, you have not tested enough".

    Remember that when all your tests pass, all you have proven is that your test cases pass and that all things you did not test are likely to fail (and probably at the worst possible moment).

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

Re: Testing in real life
by moritz (Cardinal) on Dec 26, 2011 at 06:55 UTC

    Tests suffer from the same problem as code: they can have bugs. So they undergo the same development as the code they test, and it sounds to me as though you've done everything right:

    First you tested the cases you could think of, and when you find a limitation in your approach, you expanded it.

    1. When should I write tests?

    When (or before) writing new features, and when you discover bugs.

    What should I test?

    The stuff you're working on.

    Tests are a tool to achieve a goal (here: developing and maintaining reliable software), so they should be used to best achieve that goal. If you can't think of a way that your code would make a particular test fail, it's probably useless.

    Tools like Devel::Cover can help you identify spots in your code that aren't covered, but you shouldn't blindly try to increase the coverage either.

Re: Testing in real life
by sundialsvc4 (Abbot) on Dec 27, 2011 at 21:53 UTC

    The best that a test can ever do for you is to show you where a bug isn’t ... and this only if the test itself does not contain a bug.

    However, even though this may at first sound like a Faustian quest, it really isn’t, because another thing that a good test suite can tell you is:   “it still passes.”   So, after you have made a code-change and verified that it all compiles correctly, you add to the test suite (if necessary), and re-run the whole thing, and ... “good, it still passes.”   When the tests suddenly begin to fail, they help you identify the bug while the bug is still “fresh.”   Always bear in mind that the bug could be in the code, in the test suite, or both.   (And, of course, that it is very hard to “bear in mind” that fact.)

    I like to write test suites before I write the actual code, because the test suite describes the code as I would like for it to be used and to behave.   Sometimes I realize while writing the tests that my concept for the routine in question has a design-flaw in it.   Very often while perfecting the module I discover that my implementation has a hole.   But when everything actually works, I have pretty good reason for confidence that the code is fairly close now to being rock-solid and reliable.

Re: Testing in real life
by Xiong (Hermit) on Jan 10, 2012 at 11:01 UTC
    1. When should I write tests?

    Before, during, and after writing production code.

    2. What should I test?

    Everything.

    I'm not the guy you kill, I'm the guy you buy. —Michael Clayton