Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re: Non-deterministic Testing

by tmoertel (Chaplain)
on Jan 29, 2005 at 05:43 UTC ( [id://426193]=note: print w/replies, xml ) Need Help??


in reply to Non-deterministic Testing

Even if you cannot determine whether each result returned by your algorithm is correct, you probably can determine whether the distribution of results is correct. (This is what I had to do to test the random-data generators for Test::Lectrotest.)

First, analyze your algorithm to determine the expected distribution of the results. Then run your algorithm in a large number of random trials to generate a set of observed results. Finally, use statistical tests to determine whether the results conform to the expected distribution (for the level of confidence you desire).

There is a trade off between the degree of confidence you require and the "power" of statistical tests to detect small deviations from the norm. The more confidence you want, the less power the tests will have. The way to increase the power is to run more trials. Unless you have a really bizarre distribution, however, a few seconds of compute time will probably provide more than enough data to provide ample confidence and power. (But still you should do the math to verify this for your situation.)

That is the idea. Now, to implement it, you will need to (1) determine the distribution of your expected results, (2) write a test driver to run the trials and collect the output results, and (3) write code to test the collected output against the expected distribution.

Steps (1) and (3) require some familiarity with statistics. (If you need a refresher, take a look at the resources on the Statistics on the Web page maintained by Clay Helberg.) If you are not writing unit tests to be automated, you can perform step (3) by hand in a statistics program like R. Otherwise, you will need to code up some statistical tests or borrow them (e.g., Statistics::ChiSquare) and call them from your unit tests.

I hope that this helps.

Cheers,
Tom

Replies are listed 'Best First'.
Re^2: Non-deterministic Testing
by osunderdog (Deacon) on Jan 31, 2005 at 14:29 UTC

    I'm glad someone mentioned R. It is a very powerful language. Dare I say R is the Yin to Perls Yang.

    It's also equally hard to master


    "Look, Shiny Things!" is not a better business strategy than compatibility and reuse.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://426193]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others rifling through the Monastery: (9)
As of 2024-03-28 18:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found