in reply to Re: Re: Automatic generation of tests
in thread Automatic generation of tests

It's obvious that tests do not cease (edit: typo FIXED!) to evolve after the code is complete. However, it should be stated that (if you do have a solid API) writing tests first has some advantages. Namely, you can make sure you implemented your entire API and that API works.

Once you start scrolling through hundreds of lines of code, it's hard to visualize your API use cleanly because you start to confuse the API with the implementation. Again, I don't do this nearly enough, but it has great merits and this is something I *should* do for larger projects.

By all means, tests should evolve while the program is written -- however boundary coverage should be one of the first kinds of things you think about when testing your API. That is what automated testing is for. If your API passes all tests, and the whole of your API is tested, you won't have any issues with coverage...harder said than done.

Naturally, testing code by hand is critical in validing that the tests themselves are valid.

  • Comment on Re: Re: Re: Automatic generation of tests

Replies are listed 'Best First'.
Re^4: Automatic generation of tests
by adrianh (Chancellor) on Feb 23, 2004 at 17:21 UTC
    It's obvious that tests do not evolve after the code is complete

    Depends on your definition of complete. My code collects a new test every time I discover a bug :-)

[reply]
Re: Re: Re: Re: Automatic generation of tests
by Anonymous Monk on Feb 23, 2004 at 19:10 UTC
    By all means, tests should evolve while the program is written -- however boundary coverage should be one of the first kinds of things you think about when testing your API. That is what automated testing is for. If your API passes all tests, and the whole of your API is tested, you won't have any issues with coverage...harder said than done.
    See the reply to adrianh above regarding internal/algorithmic boundary conditions.

    TDD is an *excellent* practice. Didn't I start out with that line? I am only trying to get the point across that it isn't the be-all and end-all of testing, it has limitations.

    Steve McConnell's Code Complete 2nd Ed. cites research indicating that while developers think they are getting high code coverage in testing, in reality they get 80% at best, 30% at worst, and around 60% on average. 60% is very, very, very low code coverage. The studies aren't recent, but if anything has changed I suspect it is only that developer confidence has risen, not that test coverage has.

[reply]
      The studies aren't recent, but if anything has changed I suspect it is only that developer confidence has risen, not that test coverage has.

      No the studies are not recent - and none of them are for people doing TDD. My experience, and the experience of others I've talked to doing TDD, is that code coverage goes way up when you do TDD.

      This isn't really surprising since you should not be writing code with TDD that isn't being exercised by a failing test.

      Now, if only somebody could find the time and money to do some research :-)

      (and just to emphasise that I agree completely that TDD isn't all there is to testing :-)

[reply]
        However if you start doing arbitrary rewrites of code aimed at breaking the test suite you're no longer doing TDD :-)
        This isn't really surprising since you should not be writing code with TDD that isn't being exercised by a failing test.

        No one is talking about arbitrary rewrites. Refactorings can easily result in green-bar code that wasn't immediately motivated by failing tests and that contains untested branches.

        This doesn't mean that I think more traditional white box testing, branch coverage, statement coverage, etc. are useless - far from it. They're excellent tools and you can find many bugs with them.

        Seems we aren't really in much disagreement after all. My original point was only that too often I've seen TDD adopted "at the expense" of more traditional testing methods rather than as a complementary "design" process.

[reply]

In Section Meditations