When you make all of your tests pass/fail, it's easy to write a script that summarizes the results. Very few people are going to scan several hundred (or thousand) lines of test output anyway. I don't. I just want to know what (if anything) failed. xUnit's "greebar" is a nifty binary summarizer. You either get the green bar (yeah!) or you don't (boo!). Try to summarize ad hoc tests gets messy. You're essentially trying to turn it back into a pass/fail test, so why not do that in the first place?

Even if you have a relatively small amount of test output, ad hoc results make you work for it, which runs counter to the virtue of Laziness. Let the computer do that.

As for running tests less frequently... that depends on what they mean by less frequently. I run tests several times per hour during normal development, though I usually factor tests so that slow ones (e.g., ones that go against a database) are in a separate script that gets run less often. I find that 20-30 seconds is the upper limit of where I start to get impatient and hesitate to run tests.

In general, you should only test code that's worth something to you. If you don't care whether some code breaks, don't test it. Similarly, if you don't care about the results, don't automate them.

I personally don't understand why you'd go to the trouble of writing tests if you're not going to look at the results, but some people are funny that way.

(As a side note, if you wrote your tests with Test::More, you could automate them through Test::Harness like the CPAN module does. (Appealing to authority, I wrote some of those tests.)

Similarly, if you don't care about the results, don't automate them.

I wouldn't go quite that far myself. There can be times when writing one-off tests can be very useful - e.g. Exploratory Testing (pdf).

However, in the context of unit and customer tests I agree completely.

The other one of us believes that it makes sense for the output of some tests to require a bit of eyeballing, that tests should be run less frequently and that the effort of making everything a boolean pass/fail isn't worth it.

I'm in total agreement with dws. I find it's almost always better to automate your tests. Requiring human intervention means that it takes longer to run the tests. If it takes longer to run the tests they tend to get run less often. If your tests are run less often bugs creep in.

In my experience the extra time spent dealing with bugs and eyeballing the test results will be much more than the time it would take to automate the tests.

One thing I am fond of is I use a little script to do the nifty rsync hard link diff backup on the directory that I am updating in, every few changes I make I save n run the script which (in the background) hard links the dir structure and creates a snapshot of the whole directory. After I am done for the day I will run my tests on the latest directory, if it fails I run the test down the list of directories until i find a pass then diff the last two directories. It makes finding core bugs that slip in very easy. The method to do the incs is here I use a small perl script to generate a hires timestamped dirname each rotation and vim to auto run it on save. The cool thing about it is you get all of those snapshots without much disk space cause it relys on hardlinks. I know this has little to do with your question, except adhoc tests would make something like this impossible.

-Waswas

You might want to consider running your tests even more often - maybe even every time you hit save.

That way you get to see any test failure quickly, and it's very likely to be related to a change you have just made - so the bug is even easier to track down.