http://qs321.pair.com?node_id=1113898

Amblikai has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks, this is probably going to be a very quick and easy question to answer but i'm hoping for a bit of an explanation.

I've just started to get into testing my perl programs and looking into TDD

I've read the testing tutorials and the pages for Test::More and Test::Simple etc, and i totally understand the concept (i hope).

However, one thing i'm still not sure of might be more of a best practices type question..... A scenario..

Let's say i've written a perl program with classes and methods and subroutines and all that. I have a test program which tests all the class methods and the subroutines, but what about the "glue code" in between it all in the main program?

Does that make sense? In my main program, i've created a new object say, and then i might be getting some data from stdio and manipulating it with some code before passing it to an object method. How do i test all that other code in the main program? Or am i supposed to be coding everything in subroutines?

Apologies if this is a dumb question, any help appreciated!

Replies are listed 'Best First'.
Re: A brief question about testing/best practices
by Your Mother (Archbishop) on Jan 20, 2015 at 17:25 UTC

    To amplify what choroba and davido said, unit tests would be those for your classes and methods (probably, depends on how isolated they are) and testing the “glue” as you say would be system testing. Here’s a naïve skeleton to do both that ought to help you get your head around it and some ways to organize it–

    use strictures; # strict + fatal warnings use Test::More; use Capture::Tiny "capture"; ok 1, "1 is ok"; # Test simple truthiness. diag "Dialog always shows"; note "Notes show when running verbose tests"; # Isolate a batch of tests into a subtest. subtest "My unit tests" => sub { require_ok("DBI"); # Use your own package here. # Test methods/subs in as minimalistic/decoupled a way as possible +. done_testing(1); # + whatever you add. }; subtest "My system tests" => sub { my ( $out, $err, $exit ) = capture { system qw/ ls -l -A -f /; # Your program/glue here instead. }; is $exit, 0, "Executed normally"; ok ! $err, "Without errors"; like $out, qr/\n-rw/, "Found some read/write nodes... very clumsily"; done_testing(3); }; done_testing(3);
    moo@cow[44]~/>prove pm-1113898 -v pm-1113898 .. ok 1 - 1 is ok # Notes show when running verbose tests # Subtest: My unit tests # Dialog always shows ok 1 - require DBI; 1..1 ok 2 - My unit tests # Subtest: My system tests ok 1 - Executed normally ok 2 - Without errors ok 3 - Found some read/write nodes... very clumsily 1..3 ok 3 - My system tests 1..3 ok All tests successful. Files=1, Tests=3, 0 wallclock secs ( 0.02 usr 0.01 sys + 0.04 cusr + 0.00 csys = 0.07 CPU) Result: PASS

    Further suggested reading: Re: why Test::More? (my $horn; $horn->toot;), Capture::Tiny, Test::Most.

    (Update: fixed spelling of minimalistic+decoupled in code comment.)

      Thanks for the example, i'll work through this and i appreciate the link to your other response.

      Can i use Test::Most as a replacement for Test::More at this early stage?

        Yes, but I might stick with Test::More at first just to help you keep track of what’s what and when you actually needed/used something. You should, so I say, prefer Test::Fatal to Test::Exception (imported by Test::Most).

        Update: s/Test::More/Test::Most/ in last parenthetical.

Re: A brief question about testing/best practices
by davido (Cardinal) on Jan 20, 2015 at 17:15 UTC

    The right honorable David Wheeler said it best: "All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections."

    Put everything into subroutines or methods. This even includes taking user input. If you can add a layer of abstraction around the while( <> ) {... then you can mock input by substituting your own version of that abstraction, for example.


    Dave

Re: A brief question about testing/best practices
by choroba (Cardinal) on Jan 20, 2015 at 16:53 UTC
    If you want to have 100% testable code, you have to change your coding habits. It means your main program basically becomes
    use ModuleA; use ModuleB; use Runner; Runner::run(@ARGV);

    As an intermediate step, you can also test existing programs by calling them via an IPC method and examining the output or exit status.

    لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

      Thanks, what do you mean by an IPC method?

        He means "Inter-Process Communication". Some other program calls your program, or vice versa.

        So if you have a program that is supposed to check a web service in a couple ways, you get some other second program to call yours with the various inputs needed, and then you have that second one check the output it got from each way you called the service.

        It's called "black-box testing". See my other reply.

Re: A brief question about testing/best practices
by wee (Scribe) on Jan 20, 2015 at 20:54 UTC

    Getting 100% test coverage is sometimes tricky. You can do things like test-driven development (where you design the tests first and then write code that passes them), but sometimes that's not always the best way to go about it.

    I've found that things of any marginal importance (and some of seemingly none) tend to grow organically. Someone in another group will ask if your program that does X might also do Y. Or is it possible to have your program's result saved and analyzed later? And then that program becomes a sort of automation tool. That sort of thing. It's not ideal, but I've seen it happen a lot.

    So what I do is make as many discrete parts as possible. If the thing is supposed to send an email on completion, I write a function/method that does exactly that, and only that. Then I can write a test for it, and verify that it's sending email. If I later add functionality that attaches a file or whatever, I can run my original test as well as one that tests for an attachment and verify that nothing broke.

    Contrast with just writing one monolithic procedural script. It's not very testable, and a change in one place might break a bunch of other things without you knowing about it. But if you can get yourself to the point where you know Method Foo does this thing or that thing, you can write tests that give it one or the other and check to see what it did. And then you write tests that give it neither thing it needs. And then both. And then garbage for one and valid input for the other. And so on.

    So when you're writing your initial code ask yourself if stuffing a bunch of junk in a foreach loop in your main program is better or worse than breaking that out into a (testable) subroutine. It may not be. Readability might suffer. You just have to use your best judgment.

    For all the "glue parts" that hook everything up, do a search for "white-box testing". It's where you figure out certain inputs into the main program as a whole, in such a way as to exercise specific functionality, and then you check for certain known good output. Using clever tests, you can have your glue code call quite a lot of your non-glue code. Note that you can easily design tests that will never have a real-world analogue. That might or might not be desirable.

    You can also do what's called "black-box testing". It's similar to the above, but you don't care about knowing that each possible execution pathway has been tested. You give the program input X and look for a desired outcome. This test sorta pretends to be an end user, in other words. The user does "this" and expects to get "that".

    When combined with good unit testing, one or both of those should sufficiently exercise your code as a whole enough such that you have some peace of mind.

Re: A brief question about testing/best practices (just do it)
by tye (Sage) on Jan 20, 2015 at 21:53 UTC

    Actually, it often isn't particularly hard to test complete scripts.

    Lately I've been refining an approach that lets me test scripts much more extensively. For example, I can test a script that makes modifications to the DB w/o having those modifications actually happen. A few of the test scenarios require following a few specific best practices that were already common for our scripts, mainly, writing scripts in a format like:

    #!.../perl ... use strict; ... load more modules ... our $MAX_SECS = 60; ... more configuration 'constants' ... Main( @ARGV ) if $0 eq __FILE__; return 0; sub ... ... more subs ...

    Then I have a module that facilitates testing the script by, for example, having the test script load mocks, fork, set $0, load the script (which runs it). A common mock is one that allows full access to the DB except that 'commit' only pretends to happen and you can also load temporary fake data either on top of or in place of existing data on a table-by-table basis.

    STDOUT and STDERR of the script get directed to temporary files so the test code can make assertions about what does or doesn't get output. And my module facilitates setting up what text the script should find if it reads STDIN.

    Other scenarios include:

    Load()
    Reads in the code of the script, prepends "return 0;" and '# line 1 "$file"' then eval()s it. This allows the test script to then call subroutines in the script, including Main().
    LoadAndInit()
    Just don't set $0 then load the script so the globals get initialized and just Main() doesn't get called. This scenario is the one that requires the script under test to follow that best practice.
    Spawn()
    This forks then runs the script but doesn't wait. There is even a pipe between parent and child so the child can run both "before run" and "after run" code that can even exchange data with the parent. The "before run" code runs after the script is compiled so it can override initializations and mock other things that are harder to mock with the "before compile" code of the test script.

    Yes, factor your code into modules that are easier to test. But having a bit of code that isn't factored into such doesn't mean it is that hard to test it, at least not in Perl.

    And, yes, I sometimes write tests for code before I completely refactor the code. So I have to write tests for code that isn't perfectly factored.

    Oh, also, the test scripts start with, for example, "package BinTest;" so they don't accidentally overwrite something in the script's main:: package.

    - tye        

Re: A brief question about testing/best practices
by CountZero (Bishop) on Jan 20, 2015 at 20:56 UTC
    If you want to test the whole of your application, then your application must be able to accept input from the test-code and return results which the test-code can ... test. Whether that will be possible depends entirely upon your application.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

    My blog: Imperial Deltronics
Re: A brief question about testing/best practices
by sundialsvc4 (Abbot) on Jan 20, 2015 at 18:04 UTC

    Let's say i've written a perl program with classes and methods and subroutines and all that. I have a test program which tests all the class methods and the subroutines, but what about the "glue code" in between it all in the main program?

    You need to test, not only the object in isolation, but also its interface to its client, as used by what you refer to here as “glue code.”   Tests are built-up in a sequence, from the atomic to the contextual.

    Consider, as an excellent example, the testing strategy used by the venerable DBI, or any similar very complex CPAN module.   The first tests that are run exhaustively test low-level attributes and behaviors of the object(s), including unpublished ones that you do not directly use or see.   Subsequent tests rely upon the fact that all of those more-atomic tests succeeded.   (Such that they might BAIL_OUT if not, as there is truly no point to continue.)

    The final tests in the overall test-sequence set up actual scenarios in which multiple calls are made to the object, as a client might do when that client is doing either the right-thing or the wrong-thing.   They are mimicking the “glue code.” The client should find that sequences of calls which are supposed to be made in-sequence produce the intended result, and that sequences which are not valid are, in fact, detected and rejected in the proper way.   If the contract that the object makes with its users are that certain combinations of events are invalid, then the object must enforce that contract, and so you must by testing prove that it [still] does.  

    It is frequently suggested that you should “write the tests first,” and use those tests to specify (and then, to demonstrate) what the object is supposed to do.   Now, go write the code that passes 100% of those tests, first to last, top to bottom, stem to stern.   I have tried that approach, and found that it worked.   That it yielded trustworthy code ... and that it compelled me to think the thing all the way through before I started writing.   We all know the old saw that says, “the last 20% of the code takes the other 80% of the time.”   Well, I’ve found that this technique significantly reduces that effect.   Development proceeds, at first blush, “more slowly,” yet it proceeds much more surely.

      This is excellent advice, thanks.

      With regards to writing all the tests first, i've been reading up on Test Driven Development and this seems like the core point. Does this work out in practice? I know that when i start a project i tend to jump in and start coding! Half out of excitement and half out of necessity.

      Also, how do i apply this method to existing scripts where i don't have test written?

      Are there any good documents or links you know of that show the best approach to take?

      Thanks again!

        In my experience – and I can only speak from my experience – the answer for me is:   “yes, but it’s not a religion.”   On the one hand, there is always the (legitimate) urge to “get started.”   On the other, there is the (also legitimate) concern that it takes exactly the same amount of time to write source-code that you won’t use, as it does to write source-code that you will.

        A very interesting viewpoint on this picture was described in a book, Managing the Mechanism, which so far as I know is available right now only on the Kindle.   The premise is that, when you are writing software, you are constructing a machine ... and, not just “a machine,” but “an automaton.”   A mechanical machine that has to successfully play Chess and win.   When you frame the challenge of software in that way, the importance of specification and testing becomes much more intuitive.

        I don’t reach for textbooks to tell me what to do.   When faced with a task of any significant complexity, I definitely do find a test-driven approach to be beneficial.   Why?   Because I know the value of actually being able to use myModule; with the same degree of confidence (and the same technical justification for holding such confidence ...) as I know that I am able to do when I use DBI;.

        When you use a battle-tested CPAN module such as this one, you do not give a second thought as to whether this module will actually perform as the perldoc promises you that it will.   And, if you have ever paid attention while that module and its many brethren are being installed, you will know why this is so.   You will recognize the need to be able to make the same statements and to have the same assurances with regard to “the stuff that you wrote.”

        So ... “it’s not a religion, but it is a Best Practice that can be routinely achieved.”   With a little extra effort.   Which is worth it.

        That has been my experience, anyway.   But always keep in mind that these are simply guidelines.   Strategies, if you will.   Strategies that have proven useful-enough to other people (like me), that they would recommend those strategies to others.   Your Mileage May Vary,™ but My Mileage™ so far has been pretty darned good.

Re: A brief question about testing/best practices
by Discipulus (Canon) on Jan 21, 2015 at 09:36 UTC
    thanks Amblikai to had asked that question! seriously.

    Personally i'm sticked many step behind you, i'm still approaching the Test quest and what i noticed is some lack in documentation on this subject. Not in the sense that are not explained all features of Perl's testing possilities nor we dont have good books covering the matter.

    I'm speaking about habits (as choroba said) or change of perspective testing imply. Beacuse of this i'm very happy to read Your Mother 's, wee's and tye's answers.

    I think Test and the mind change it implies is a crucial breakpoint in the Perl learning curve (i'm still before such breackpoint) and tell the difference between who learned the language and who can use it effectively and, shall say, profesionally.

    I would like to see a red warning in all Perl Test docs like:Warning: testing an existing program not planned to be tested is a quite diccult task you should avoid.

    In such panorama I'had found very intriguing the Modulino idea by brian_d_foy.

    As major shift of habit, you can think about your code in a 'distribution' way; not only tests but docs and installation facilities too. This multiply the workload by a factor 3 but then you have really good code, maintenable and usable by others.

    I'll like to see more example and answer in this thread so we can point other monks to it in the future.
    L*

    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.