Do you ever feel that your testing is a sort of tautology (A is true because, well, A is true)? If so, how do you put in place checks and balances to avoid this?

Recently I've been working on a test suite that validates the data and behavior belonging to a number of big ugly complex objects (hence forth called BUCOs). These objects are built up of smaller pieces and there are unit tests for each of these smaller pieces.

To make my testing easier I've try to build up a repository of pre-built pre-tested objects, stored in Perl constants. Then I compose the BUCOs from these smaller (already tested objects). This works nicely most of the time because the BUCOs are usually just composite objects and the constructor takes the components as parameters.

However, not all BUCO's can be built this way. Some of my BUCOs interact with the file system or a database or some other major system. In each of these cases my practice is to create a mock environment. For example, if I need to see how a BUCO interacts with a directory tree, I might use File::Temp to create such a tree complete with files and subdirectories and any other goodies I need in this mock environment. The BUCO constructor often needs some sort of identifiers from this mock environment. The expectations also need to be derived from this environment.

When I build test objects I like to hard code as much as possible of my expectation set. Human error can be easily introduced when expectations are hard coded, but at least the expectations are out there in the open to be inspected. However, when constructor inputs and expectations are drawn from a mock object, then hard coding isn't always an option. For example, if an expectation concerns a fully qualified directory path and the root of that directory is a randomly generated temp file name, then one has no choice but to generate the fully qualified path from the temp root and a relative path.

The problem I repeatedly run into is this: oftentimes the code used to extract an expectation from the mock environment looks suspiciously like the very code I'm testing! I try various things to avoid a tautology, but I don't have a lot of confidence in any of them:

• alternate algorithm - use an alternate algorithm to extract the data, preferably one that uses more hard coded data than the algorithm being tested. The problem I have with this approach is that often the code being tested is fairly simple in and of itself. It is not always easy to find an reasonably simple alternate algorithm. Yet the test shouldn't just be skipped - we still need to make sure that the original code doesn't have some sort of dumb typo in the "simple algorithm". (It is amazing how common typo type bugs are in the early testing rounds).
• recode from spec - the problem here is that I might be testing my own code. In that case there is a risk that I will repeat my own mistake. Or there might be an easy logic trap that both I and the original coder fall into. In either case, recoding from spec is no guarentee of correctness.
• lowered expectations - Give up and decide that conformance testing for certain kinds of BUCOs can never be more than a tautology. For these BUCOs tests exist only for regression purposes (i.e. that past behavior will be maintained as the code changes) and as verification that using a function doesn't crash the application.

So what safeguards do you put in place to avoid tautologies when testing BUCOs? Is there an approach I'm missing? Are some things just untestable? I would be most grateful for the experiences and opinions of my fellow monks.

Thank you in advance, beth

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.