|
in reply to Testaholics Anonymous (or how I learned to stop worrying and love Test::More)
I would just like to say that redundant tests have some negative qualities:
- The test suite will run slower, and with 2,500 tests, that is going to become an issue.
- Time spent developing unnecessary tests detracts from development of new code.
I have some common sense guidelines for test writing:
- Test the boundary cases far more than the regular cases.
- Focus tests on buggy or expected-to-be-buggy code.
- If you can write the test cases first, then good for you, but be aware that the test suite is broken until the code is completed.
- Interesting things happen when Person A writes tests for Person B's code.
- If you have performance requirements, write tests that only pass if those requirements are met.
- Coverage is important.
|
Re: Loose Guidelines
by stvn (Monsignor) on Mar 31, 2004 at 21:13 UTC
|
pbeckingham
Excellent guidelines. Especially the "Person A writing tests for Person B's code" one. ++
The test suite will run slower, and with 2,500 tests, that is going to become an issue.
To be honest, I am not really concerned about this, since its a rather large framework (almost 150 classes, almost 13,500 loc) and (for us anyway) its the foundation of our applications. It really has to be reliable for us, so even if the tests took an hour to run that would be okay for us. I realize this may not be okay when we get around to distributing this, but I will deal with that when the time comes.
Time spent developing unnecessary tests detracts from development of new code.
Thats just the thing, the tests run are redundant, but alot of them are in functions and therefore really didnt take anymore time to develop. Its actually saving me time, since I can re-use the test even though it is somewhat testing something I know is already tested (sort of).
-stvn | [reply] |
|
|
To be honest, I am not really concerned about this, since its a rather large framework (almost 150 classes, almost 13,500 loc) and (for us anyway) its the foundation of our applications. It really has to be reliable for us, so even if the tests took an hour to run that would be okay for us.
Nice in theory. Sucks in practice. More than one developer here (on a "must never fail on pain of death" project) has skipped running the entire 1.5 hour test suite because "their one little change" won't break things and their one little test didn't break. When you're in a hurry, one and one half hours can seem like a long time :)
Redundant tests are bad. They slow things down, they are more tests to maintain and they provide no benefit. You don't need to prove that 2+2 == 4 more than once.
| [reply] |
|
|
Nice in theory. Sucks in practice. More than one developer here (on a "must never fail on pain of death" project) has skipped running the entire 1.5 hour test suite because "their one little change" won't break things and their one little test didn't break. When you're in a hurry, one and one half hours can seem like a long time :)
I see your point. We have institued a "must run tests under pain of death before commiting to CVS" policy, but even that can be subverted by modifying the working directory. We have several distinct sub-systems which do not interact with one another (except in maybe in application code that uses the sub-systems), I suppose I could use the prove utility that comes with Test::Harness (my other new favorite toy along with Devel::Cover), to script sub-system tests in some way.
Redundant tests are bad. They slow things down, they are more tests to maintain and they provide no benefit. You don't need to prove that 2+2 == 4 more than once.
Agreed, but what about if they are redundant because they are in a function? Here is an example:
sub test_Base_Interface {
my ($o) = @_;
can_ok($o, 'new');
can_ok($o, 'helloWorld');
}
sub test_DerivedFromBase_Interface {
my ($o) = @_;
can_ok($o, 'new');
can_ok($o, 'gutenTag');
test_Base_Interface($o);
}
The can_ok($o, 'new') test is repeated (because 'new' is overridden in 'DerivedFromBase'). Its redundant, but "for a reason" (maybe not a good one). Is this still bad? Or am I being silly and should just use Test::Class or something similar?
-stvn | [reply]
[d/l]
[select] |
Re: Loose Guidelines
by dragonchild (Archbishop) on Apr 01, 2004 at 01:18 UTC
|
The test suite will run slower, and with 2,500 tests, that is going to become an issue.
I have serious issues with this line of thought. In my opinion, one should run every test that deals with any piece of code that has a 0.00001% chance of dealing with the code you changed since the last time you ran a full test suite.
Now, my codebase is broken up into 23+ Perl distributions (each its own CVS project) and 4 other CVS projects containing Oracle DDL, templates, and the like. The projects have a definite hierarchy (which is delineated in every distributions Makefile.PL). If I change some code in XXX::Foo and YYY::Foo depends on XXX::Foo, I need to run the tests in both XXX::Foo and YYY::Foo. Period. There's just no way around that.
Now, I don't have to run the tests for XXX::Foo if YYY::Foo changes, and the test suites need to be flexible enough to handle that. But, I think it's foolish to complain about test suite size or the time to run it. You have tests for a reason. Don't sabotage yourself just because you're impatient. It may be a virtue, but so is patience.
------
We are the carpenters and bricklayers of the Information Age.
Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose
| [reply] |
|
|
Please bear in mind that I was referring to redundant tests slowing the test suite. Redundant is being used here to mean unnecessary. If you are advocating more tests, each of which increases coverage to some degree, then I am in agreement with you.
Redundancy in the test suite - by definition - is pointless, and serves to lengthen the wait every time it is run, and that will multiply up and become, as I said, an issue.
| [reply] |
|
|
