in reply to Test output: Interpret or pass/fail

The other one of us believes that it makes sense for the output of some tests to require a bit of eyeballing, that tests should be run less frequently and that the effort of making everything a boolean pass/fail isn't worth it.

I'm in total agreement with dws. I find it's almost always better to automate your tests. Requiring human intervention means that it takes longer to run the tests. If it takes longer to run the tests they tend to get run less often. If your tests are run less often bugs creep in.

In my experience the extra time spent dealing with bugs and eyeballing the test results will be much more than the time it would take to automate the tests.

  • Comment on Re: Test output: Interpret or pass/fail

Replies are listed 'Best First'.
Re: Re: Test output: Interpret or pass/fail
by waswas-fng (Curate) on Aug 07, 2003 at 06:30 UTC
    One thing I am fond of is I use a little script to do the nifty rsync hard link diff backup on the directory that I am updating in, every few changes I make I save n run the script which (in the background) hard links the dir structure and creates a snapshot of the whole directory. After I am done for the day I will run my tests on the latest directory, if it fails I run the test down the list of directories until i find a pass then diff the last two directories. It makes finding core bugs that slip in very easy. The method to do the incs is here I use a small perl script to generate a hires timestamped dirname each rotation and vim to auto run it on save. The cool thing about it is you get all of those snapshots without much disk space cause it relys on hardlinks. I know this has little to do with your question, except adhoc tests would make something like this impossible.

    -Waswas
[reply]

      You might want to consider running your tests even more often - maybe even every time you hit save.

      That way you get to see any test failure quickly, and it's very likely to be related to a change you have just made - so the bug is even easier to track down.

[reply]
        I used to, this way gives me three advantages.

        1. I tend to work more efficantly when I can stay on track -- If I have an idea how to improve something and get side tracked on fixing a bug that just poped in the code then I take a while to get back into the flow once I fix it. If my bugs were usually severe and requiring a complete redesign this would be moot.

        2. If I do implement solution X, and in the middle of implementation I figure out that solution Y is better for whatever reason I have many stop points I can jump back to implement solution Y.

        3. I can audit my code almost change by change and show movment.

        -Waswas
[reply]

In Section Seekers of Perl Wisdom