If this were me confronted with this project, I'd strongly consider going down the 'Test' path in parallel with the 'Automation' path. There will be enough cross-talk between the two to make this a very useful approach. The Test path will give you the means to determine and report your C++ code in action, which the Automation path can use to make go/stop and fix/nogo decisions.

Test scripts intended for the Test::Harness framework can be talked into handling all of what you require; they are, after all, just Perl programs.

Basically, you're looking at using Conditional tests like Test::More supports. It works like this:

SKIP: {
   skip $num_tests, "Wasn't able to start the 'foo' daemon" if (
      try_start('/bin/foo') or wait_for('Please start foo', '5m');
   );

   # ... the tests that depend on foo
}
[download]

If you need to bail out completely, you can use the BAIL_OUT method provided by Test::More:

eval {
   try_start('/bin/foo') or wait_for('Please start foo', '5m');
   die "Unable to find a running foo" unless is_running('/bin/foo');
};
if ($@) {
   BAIL_OUT($@);
}
[download]

Please note that these are pretty high-level suggestions, but I hope they can start you thinking in the right direction.

I did very much what you are describing, wrote the automation steps first and the tests later. Also, the distinction you make between steps failing and tests failing is one that I've followed.

Probably the most significant thing I found was that the automation steps ended up being reused for all sorts of things - in fact, the test framework is not in use yet, but I've written several support tools (for component & log file monitoring, enabling/disabling components etc) using the automation classes which save me a lot of work from day to day. Since the automation classes also get much more use this way, they've improved considerably in error detection & recovery in ways that they might not if they had only been used for testing.

For that reason I'd recommend not coupling your UI with the automation - instead have packages containing automation functions that return success or failure, and write the user interaction code separately - possibly as part of your test script.

I also had a similar problem of convincing other people this was a good way forward - I got around that by designing the whole thing up front, then initially only implementing those classes I needed for a simple demonstration test case to get buy in.

Much of what you need is built into perl (backticks to run command line utilities, normal filehandling for reading log files, pattern matches for spotting log messages etc) - which is one reason why perl is ideally suited to the task.

If you are going to have a very large volume of tests and associated test data, I'd suggest keeping the test and it's associated test data together for maintainability. I initially did this by storing them both in a database, but later found that XML files were more flexible and easily maintained. If you use the Test::Harness framework for your tests (and I see no reason not to - I didn't really have the option) you could perhaps put the test data for each file in a __DATA__ block, or in a separate file with the same base name, or some other mechanism, as long as there is an easily spotted connection between them.

Edited by planetscape - removed link to copyrighted content

Considered by grep: Reap: Copyrighted Material
Unconsidered by Arunbear: keep (and edit) votes prevented reaping (Keep: 1, Edit: 2, Reap: 9)