Your skill will accomplish what the force of many cannot |
|
PerlMonks |
Test harness for executablesby sfink (Deacon) |
on Nov 26, 2008 at 19:20 UTC ( [id://726194]=perlmeditation: print w/replies, xml ) | Need Help?? |
While I'm between jobs, I've been trawling through various old code that I've written to see if there's anything I could clean up and release. One thing that I have found particularly useful is a handrolled test harness I wrote to make it easier to create unit tests for a program I was working on.
I don't think there's anything stunningly novel about it, but I found it to strike a good balance between capability and simplicity. It was pretty easy for other people to pick up and start writing tests within, and for diagnosing the inevitable test failures. I wrote it before the recent TAPification of Perl test tools, so it could probably make use of some of the newer modules. (I wrote this tool partly in response to the difficult of extending the older Test::Harness to do what I needed.) Enough introduction. Why should anyone care? I'll try to enumerate what seems to be unique or vaguely unusual about this tool:
NAMEharness.pl - Test harness for running test files through an executable application
USAGEperl harness.pl --binary=/PATH/TO/APP --tests=SPEC foo.t The test specification is a comma-separated list of test numbers or ranges. For example: perl harness.pl -t 2,5-8,20- mytest.t will run the tests given in the 'mytest.t' test configuration file numbered 2, 5, 6, 7, 8, and every test starting at 20.
CONFIGURATION SYNTAXConfiguration file syntax looks something like: default { runtime = 2 } return_status_zero { outcome = ok } new_version { min-version = 3.2.7 } test : return_status_zero, new_version { input = mytest/basic.xml desc = Basic test output { 0: /correct/ 1: ! /Memory leak/ } } The configuration file is describing a hierarchical structure of named keys and their corresponding values. The above could be written (almost) equivalently as default.runtime = 2 return_status_zero.outcome = ok new_version.min_version = 3.2.7 test.input = mytest/basic.xml ...etc... A section with a name followed by an open curly bracket defines a "scope", meaning that every key seen within the body of the curly brackets is assumed to be prefixed with the path of that name. Sections introduced with "section : parent1, parent2, ... {" do the same, except they also copy all of the key/value pairs from both 'parent1' and 'parent2' into that section. So that first example is also (almost) equivalent to: test.outcome = ok test.min-version = 3.2.7 test.input = mytest/basic.xml ...etc... This syntax is provided so that you can have a large number of tests that are almost the same, but without repeating information for every one. The section named 'default' is always inherited by everything, so it may be used for global configuration (eg, the entire test file should only be if the version is at least X.Y). The only "special" section name is 'test'. 'test' is immediately expanded into 'test-NUMBER', so that each test is unique. (If you reused any other name, it would happily add to that section.) The above description is completely generic, and says nothing about how tests are actually run.
TESTSEach of the 'test-NUMBER' sections specifies a test to be run. Normally, this means to invoke the application on whatever the key 'test-NUMBER.input' is set to, but if no input is given (or the special flag 'test-NUMBER.use-previous-run' is set to a true value), then the output of the previous run will be used instead. Tests are primarily specified by the name of the input to run and a description of that test. The description is only used for human-friendly output, and is not required (it will default to something like "test at mytest.t:87"). It is really just for documentation. The following keys are used to specify how to run a test spot:
The following keys are used to control whether or not this test should actually run:
The following keys are used to describe what is actually being tested:
OUTPUT PROCESSINGThe 'output' key can be set to a plain string, which will then be expected in the output. More often, 'output' can be /regex/ (a regular expression surrounded by slashes), or m!regex! (same thing, but more convenient if what you are looking for itself contains slashes.) Each line of the output will be scanned for the regular expression, and at least one occurrence must be found for the test to pass. If you want to check for something that could span lines, use the /s modifier: test { output = /initialized.*awakened/s } In this case, the regex will be applied to the entire output, not just a line at a time. (Any other Perl regex modifiers can be added in the same way, but /s is treated specially.) Finally, you can prefix the output value with an exclamation point to say that the test passes only if the regex does NOT match: test { output = ! /bad stuff/ } If you want to test multiple expressions at the same time, you have two choices: (1) have separate tests that use the same run of the reactor, or (2) make output into its own key/value section: output { 0: /foo/ 1: /bar/ bleh: /blort/ }
Back to
Meditations
|
|