Coming up with tests wasn't easy. The stupid ones that exercise your handling of Getopt::Long were simple. Getting tests that actually exercised code that deals with abstract values ...

I've been writing test driven code (no caps) for years. It started when doing functional verification of the OS/2 apis. 80% percent of the code I wrote for the 3 1/2 years I was involved had only one purpose, to test the apis it used. Writing small sections of code and the test(s) to verify it, as a single unit of work, was the only way to do it. It's one of those things that worked so well for me that I've continued doing it (for the 10+years since) even when it wasn't a strict requirement.

I still find it easier (and equally effective) to write the code that uses the interface, and then devise the mechanism for testing it. I feel that this means that I don't tailor my use of the inteface to fit my testing methods which I have seen happen when doing it the other way around. For any given project, half a dozen or so verification techniques start to repeat and it rapidly becomes second nature to decide which one fits the bill for a particular situation.

This may be less necessary for Perl than C/C++ as the availability of eval means that you can do things in perl that are simply unavailable in C(++), I still think that you need more than the simple ok/nok for most things.

This is key because for any non-trivial test, you're going to have (potentially) non-deterministic output...

One well-used technique with functions that give contiguous ranges of output for discrete sets of input is to look for and validate the transitions across the boundaries, also called edge cases (and corners cases). The trick here is to evolve your tests from the inputs rather than the outputs. That is to say, don't look for where you think (or the code under test pre-determines) that the transitions occur, but rather

• Start by iterating over the full ranges of each the inputs (initially in large steps) and look for transistions.
• Gradually reduce the size of the steps for one of the inputs whilst reducing the range(s) covered by that input. In this way you can sort of binary step your way to as small a range and the minimal steps size possible to cover the transition(s) from good to bad.
• Do this for each of the inputs in turn, returning the full range/ large steps for the previously resolved input whilst you work on the next.
• Finally, reduce the testcase to covering only the minimum set of range(s)/ step sizes for each of the inputs required to cover the transitions.

That probably sounds horrendously laborious, but for most functions (as in subroutines rather than the mathematical usage), even those with relatively large numbers of inputs (parameters), varying one parameters at a time over limited range(s) whilst iterating the others through the full spread in large steps means that you can usually quickly zero in on the transition points. You sometimes find that when you start varying the second input in smaller steps, that it isolates more (or moves) transitions in the first range, but doing things stepwise usually makes obvious when this is happening and how to compensate.

The idea of reducing the coverage of testcases is an anathema in some circles, but by concentrating on an intelligently selected subset of all possible tests, you can increase the effectiveness of the testing whilst reducing the test time. The shorter the test time, the more frequently it is likely to be run.

As with many things in life, more does not necessarily equal better. Sometimes more is just more, without benefit.