...by concentrating on an intelligently selected subset of all possible tests, you can increase the effectiveness of the testing whilst reducing the test time. The shorter the test time, the more frequently it is likely to be run.
To an extent I agree, but not totally. For most of the modules I've written the past year or so, I've found that I actually need tests for 3 contexts:
- 1. Does it seem to work in the user's environment?
- This is the test that you want typically run in a CPAN(PLUS) environment when a user installs a module. It should be relatively throrough, but shouldn't take too long.
- 2. Does it survive stress testing?
- For most linear programs not really an issue, but if you're working on threaded programs like I have the past year, you know something may appear to work under "normal" situations, but will break down when being hit with everything you got. ;-(
- 3. Do all possible combinations of parameters work?
- This is the test that you have internally to make sure that whatever combinations of parameters that can exist, that something valid happens. This is especially important with combinations of command line parameters. This can run as long as you need: for one project I'm working on, this typically runs for 45 minutes: not something you would like your average user to go through.
Most of my modules on CPAN have at least tests of category 1. Some of them have in category 2 as well. Category 3 tests I only run internally or by a user who is experiencing specific problems, not during a standard test/install sequence.
Liz