The previous sentence says “the same series of N challenges will be presented to each solution. The set of challenges may be hand selected, random, or some combination of both.” Obviously the next sentence refers to the number of solutions among those. Also, MidLifeXis will use his probability-fu to make sure the result is quite probably a good estimate of what you'd get from testing all solutions.

Okay, that makes sense.

That said, it still seems like a complicated scoring mechanism. I offer:

my $score = ( $totalTries - $minimumTries ) 
         * ( ( $totalTime - $timeInCheckSub ) / $totalTries );
[download]

With the number of tries for any given attempt being set to the theoretical maximum for any erroneous solution returned.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP PCW It is as I've been saying!(Audio until 20090817)

I recommended the score being simply the total number of tries, with the restriction that each program is tested with the same multiset of solutions. I'd now like to augment this with the condition that there be only a few (say three) different (v, p) pairs tested (in the scoring runs at least), and the score and winner computed separately for each pair, because the submissions can behave really differently for each pair, so there's no fair scoring algorithm that works. (I also suggested elsewhere in this thread that these few parameter pairs be published in advance.)

(And yes, I want to write a submission, I even have an idea, but I don't have much time on my hands now because of, you know, Real Life occupations. Sorry.)

(Update: I'm not saying I'll write a well-scoring solution or have a good idea, I just mean some working submission and some idea.)

Now we'll only need more solution proposals, or else all this meta discussion will remain rather academic :-)