In an attempt to get around the problems with QA described in The purpose of testing, I have been working on something very similar. The chief difference between what you are describing and what I've been working on is that I am seeking to generate automated tests and formal test scripts (in MS-Word) from the same metadata.

I did very much what you are describing, wrote the automation steps first and the tests later. Also, the distinction you make between steps failing and tests failing is one that I've followed.

Probably the most significant thing I found was that the automation steps ended up being reused for all sorts of things - in fact, the test framework is not in use yet, but I've written several support tools (for component & log file monitoring, enabling/disabling components etc) using the automation classes which save me a lot of work from day to day. Since the automation classes also get much more use this way, they've improved considerably in error detection & recovery in ways that they might not if they had only been used for testing.

For that reason I'd recommend not coupling your UI with the automation - instead have packages containing automation functions that return success or failure, and write the user interaction code separately - possibly as part of your test script.

I also had a similar problem of convincing other people this was a good way forward - I got around that by designing the whole thing up front, then initially only implementing those classes I needed for a simple demonstration test case to get buy in.

Much of what you need is built into perl (backticks to run command line utilities, normal filehandling for reading log files, pattern matches for spotting log messages etc) - which is one reason why perl is ideally suited to the task.

If you are going to have a very large volume of tests and associated test data, I'd suggest keeping the test and it's associated test data together for maintainability. I initially did this by storing them both in a database, but later found that XML files were more flexible and easily maintained. If you use the Test::Harness framework for your tests (and I see no reason not to - I didn't really have the option) you could perhaps put the test data for each file in a __DATA__ block, or in a separate file with the same base name, or some other mechanism, as long as there is an easily spotted connection between them.

--------------------------------------------------------------

"If there is such a phenomenon as absolute evil, it consists in treating another human being as a thing."
John Brunner, "The Shockwave Rider".

In reply to Re: Automated regression testing by g0n
in thread Automated regression testing by rinceWind

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.