McMahon's scratchpad

Public Scratchpad

I do what we call "functional testing" or "system testing" or "end to end testing" or "black box/grey box testing". When I talk about a "test harness", I mean a system that allows the user to design test cases that the harness will run and validate automatically.

The best ones I've seen (or built, in one case) drive a system not from the UI, but directly from a set of data like a spreadsheet.

First, though, I'm going to stipulate that a "test case" must contain 3 elements: an objective, a procedure, and an expected result. Only the last two can be automated.

So a trivial test plan would consist of test cases that have objectives:
1) Insert service order
2) Change service order.
3) Change service order multiple times.
4) Delete service order.

So we'll keep the data for the tests in a simple spreadsheet:

ID    Action  Data    
TC-1  I       5551212  DEMERPHQ 100 HAUPTSTRASSE
      ER      5551212  DEMERPHQ 100 HAUPTSTRASSE
TC-2  C       5551212  DEMERPHQ 200 HAUPTSTRASSE
      ER      5551212  DEMERPHQ 200 HAUPTSTRASSE 
TC-3  I       5551213  DEMERPHQ 100 HAUPTSTRASSE 
      C       5551213  DEMERPHQ 200 HAUPTSTRASSE 
      C       5551213  FOOBAR   200 HAUPTSTRASSE 
      ER      5551213  FOOBAR   200 HAUPTSTRASSE
TC-4  D       5551212 
      ER      NULL
[download]

For each action, the harness will check the database and report PASS or FAIL depending on whether what's in there matches the values in "ER", the Expected Result. Note that this lets a business user design tests without knowing the underlying implementation of the tests. (BTW, a database query may not be the best way to describe Expected Results. The most complex harness I ever built did a massive diff on sets of files, rather than a surgical query for data.)

Some other considerations: for tests like these, it is imperative that the test environment be resettable to a known state before the tests run. Part of the harness is a controllable system.

So that's an overview of what I know about building test harnesses.