http://qs1969.pair.com?node_id=270342


in reply to Testing by Contract

A naive probabilistic approach to the problem of testing all possible combinations:

Instead of testing all possible combinations, we could try to estimate the total number of bugs by a probabilistic approach. Given a list of all possible combinations, we randomly select two subsets of combinations for Tester One and Two to test.

Let T be the total unknown number of possible bugs associated with all combinations.
Let A be the number of bugs found by Tester One.
Let B be the number of bugs found by Tester Two.
Let C be the number of bugs found by both Tester One and Two.
Hence (let P(X) be probability of X)

P(A and B) = P(C)     (by definition)

  P(A)P(B) = P(C)     (independence assumption)

  A     B     C
 --- * --- = ---
  T     T     T

      A*B
     ----- = T
       C
That means, the less bugs both Tester One and Two found at the same time, the more likely there're still a large number of unknown bugs yet to be found.

Or, the more common bugs found by both Tester One and Two, the more likely that they have found most of the bugs.


_________
If someone wanna see the Venn diagram: 
   +----------------------------------+
   |                                  |
   |    +------------+                |
   |    |            |        T       |
   |    |    A       |                |
   |    |            |                |
   |    |     +------|-------+        |
   |    |     |  C   |       |        |
   |    +-----|------+       |        |
   |          |          B   |        |
   |          |              |        |
   |          +--------------+        |
   |                                  |
   +----------------------------------+

Replies are listed 'Best First'.
Re: Re: Testing by Contract
by BrowserUk (Patriarch) on Jul 01, 2003 at 11:51 UTC

    Stats ain't my forte, but wouldn't this only be true if tests A & B were both capable of detecting all the possible unknown bugs?

    If this the case, then all you need to make this work is a sure fire way of designing tests that are guarenteed to be capable of detecting all possible bugs. Actually, you would probably need a way of designing two independant tests capable of detecting all possible bugs, as I doubt it would work if the two tests were the same:)

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller


[reply]
      Since A and B are only random subsets of all possible combinations (in the case of integration testing), they are not going to detect all possible unknown bugs

      The key lies on C, the common area. If you look at the Venn diagram and if you imagine squeezing the superset T smaller and smaller, it will be less likely for C to be small--A and B must tend to overlap.

      The whole point is to estimate the total number of bugs without having to go through an exhaustive testing.

      Of course, that simple estimate probably won't be statistically very valid, since bugs are not independent. But it still gives a good conceptual insight--if a bunch of independent testers tend not to find common bugs, there're probably still pretty of bugs out there.
[reply]

In Section Seekers of Perl Wisdom