Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Re: Testing by Contract

by chunlou (Curate)
on Jun 30, 2003 at 22:10 UTC ( [id://270342]=note: print w/replies, xml ) Need Help??

in reply to Testing by Contract

A naive probabilistic approach to the problem of testing all possible combinations:

Instead of testing all possible combinations, we could try to estimate the total number of bugs by a probabilistic approach. Given a list of all possible combinations, we randomly select two subsets of combinations for Tester One and Two to test.

Let T be the total unknown number of possible bugs associated with all combinations.
Let A be the number of bugs found by Tester One.
Let B be the number of bugs found by Tester Two.
Let C be the number of bugs found by both Tester One and Two.
Hence (let P(X) be probability of X)

P(A and B) = P(C)     (by definition)
P(A)P(B) = P(C) (independence assumption) A B C --- * --- = --- T T T A*B ----- = T C
That means, the less bugs both Tester One and Two found at the same time, the more likely there're still a large number of unknown bugs yet to be found.

Or, the more common bugs found by both Tester One and Two, the more likely that they have found most of the bugs.

If someone wanna see the Venn diagram:
   |                                  |
   |    +------------+                |
   |    |            |        T       |
   |    |    A       |                |
   |    |            |                |
   |    |     +------|-------+        |
   |    |     |  C   |       |        |
   |    +-----|------+       |        |
   |          |          B   |        |
   |          |              |        |
   |          +--------------+        |
   |                                  |

Replies are listed 'Best First'.
Re: Re: Testing by Contract
by BrowserUk (Patriarch) on Jul 01, 2003 at 11:51 UTC

    Stats ain't my forte, but wouldn't this only be true if tests A & B were both capable of detecting all the possible unknown bugs?

    If this the case, then all you need to make this work is a sure fire way of designing tests that are guarenteed to be capable of detecting all possible bugs. Actually, you would probably need a way of designing two independant tests capable of detecting all possible bugs, as I doubt it would work if the two tests were the same:)

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller

      Since A and B are only random subsets of all possible combinations (in the case of integration testing), they are not going to detect all possible unknown bugs

      The key lies on C, the common area. If you look at the Venn diagram and if you imagine squeezing the superset T smaller and smaller, it will be less likely for C to be small--A and B must tend to overlap.

      The whole point is to estimate the total number of bugs without having to go through an exhaustive testing.

      Of course, that simple estimate probably won't be statistically very valid, since bugs are not independent. But it still gives a good conceptual insight--if a bunch of independent testers tend not to find common bugs, there're probably still pretty of bugs out there.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://270342]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others learning in the Monastery: (4)
As of 2024-04-23 00:22 GMT
Find Nodes?
    Voting Booth?

    No recent polls found