in reply to OT: TDD question

Getting 100% coverage is of dubious utility. I'm usually happy if I'm >85%.

A while back, I was writing an important new application for my company. It would often send out e-mails, and I had coded the mail-sending module to send e-mail via our mail server, with the user's e-mail address on the To and/or From lines. While testing it, we had always used e-mail addresses within our own domains.

On release, no e-mail was being sent. I soon figured out why: the mail server considred the web server an external host, and since it was (correctly) not configured to be an open relay, it would drop the mail when the domain portion of the From address wasn't one of our domains.

If we had been using TDD (which we aren't due to political reasons), we potentially could have shown 100% coverage with Devel::Cover and never caught this bug. The e-mail addresses used in the tests would have almost certainly been internal ones. Only a developer experianced with this Gotcha would have caught it before release.

"There is no shame in being self-taught, only in not trying to learn in the first place." -- Atrus, Myst: The Book of D'ni.

Replies are listed 'Best First'.
Re^2: OT: TDD question
by stvn (Monsignor) on Dec 06, 2004 at 04:48 UTC

    100% code coverage only means all your code has been run, not that it has been tested. There is a huge difference.

    I read a good paper called "How to misuse Code Coverage" a while ago that discusses exactly your point. It is available here as a PDF. Its worth a read if you haven't read it already.

    Personally, I shoot for 95% and above and I am happy. As I described above, I try not to worry about coverage until the end, that way I am concentrating on testing my code and not covering it. And lets not forget too that unit-tests only test isolated chunks of code. Integration tests are just as important (if not more).

    -stvn
[reply]
Re^2: OT: TDD question
by kscaldef (Pilgrim) on Dec 06, 2004 at 04:32 UTC

    On the contrary, this is not an argument for permitting less than 100% coverage, it's a argument for requiring greater than 100%. If you only exercise each line of code once in your tests, then you almost certainly don't have enough tests.

    Your story is just a special case of not testing with enough inputs to your code. A developer wouldn't have to be familiar with this specific feature of mail servers in order to catch this in unit tests.

[reply]
      ", it's a argument for requiring greater than 100%."

      Let's not be silly. Greater than 100% coverage is impossible. What this is saying that EXTERNAL factors cannot always be covered with TDD... and unfortunately most of my code deals with external factors. Not just black box code, but remote systems and odd code combinations. Firmware. Drivers. Network issues.

      TDD is great for Perl modules and small pieces. It does not adapt well to large scale systems. There you need custom automated test environments that are very domain specific and also require lots of setup and specific hardware configurations (and variations on those). Even then, it's not perfect.

[reply]

        You've taken that phrase out of context and as a result missed the point, which was that you need to exercise each line of code many times to have a serious hope of finding all or most of the bugs. Just hitting 100% by creating 1 test case for each branch is almost never sufficient.

        When it comes to your argument about external factors, Perl actually has some advantages over more static languages. Need to test rare conditions in a socket connection? Just override the relevant methods of IO::Socket to produce them when you want. The same trick works for lots of things which are otherwise hard to test.

        TDD can work for large systems, because what you do is test the hell out of the small pieces, with the same usage patterns as your full application. This doesn't replace large-scale integration testing, but that doesn't mean that you shouldn't do it. I catch lots of stuff in unit tests that don't get caught in our full-system functional tests (and vice versa).

[reply]

In Section Meditations