BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:

I have a very long running process that is testing randomly generated scenarios. At any given point in time I have two knowns: $T is the total number run so far; and $S is the number of those that have been successful. From those I can easily indicate the current success rate as a percentage.

I'm looking for some mechanism to indicate whether the processing is nearing -- or tending towards -- some level of completion.

If I introduce a third variable, target success rate, $TSR (say: 90%), then what further statistics can I derive that would give me a feel for how the process is progressing?

For example, it would be possible to calculate a number of further tests that would need to be run, that -- assuming they were all successful -- would raise the success rate to the target value. But assuming the current success rate is less than the $TSR -- otherwise I wouldn't be posting -- that would be a naive statistic.

So then I thought about trying to calculate the number of addition tests required to achieve the $TSR, assuming the current success rate continued; but of course, that will never happen.

So then I thought about trying to calculate the trend in the current success rate; and if that trend shows a consistent decline, or reaches a point where the $TSR can never be reached, then I abandon processing with the current parameters and move on to the next set.

I don't have any code because I'm just starting to think about the problem; and so far, I have got a clue what to code.

If any of this sparks any thoughts, possibilities or suggestions; please post.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re: Instrumenting a genetic algorithm.
by Arunbear (Prior) on Aug 19, 2016 at 21:40 UTC
    (Coincidentally) I've been doing some experiments with AI::Genetic::Pro. Some sample output showing the best, average and min fitness of each generation. 
    2016-08-19 21:42:34,960|GEN ==> 8, 62.000000, 18.000000, -216.900000
2016-08-19 21:43:11,714|GEN ==> 9, 62.000000, 19.000000, -28.340000
2016-08-19 21:43:49,558|GEN ==> 10, 62.000000, 26.000000, -63.970000
2016-08-19 21:44:26,772|GEN ==> 11, 62.000000, 29.000000, -47.570000
2016-08-19 21:45:03,968|GEN ==> 12, 62.000000, 37.000000, -32.000000
2016-08-19 21:45:41,308|GEN ==> 13, 62.000000, 51.000000, -34.000000
2016-08-19 21:46:18,268|GEN ==> 14, 62.000000, 56.000000, -34.750000
2016-08-19 21:46:55,695|GEN ==> 15, 62.000000, 58.000000, -2.000000

    [download]
    The difference between the best and average fitness shows the GA is converging, and it may be a good time to stop.
[reply]
[d/l]
Re: Instrumenting a genetic algorithm.
by SuicideJunkie (Vicar) on Aug 19, 2016 at 16:00 UTC

    Perhaps you might start with a chart showing the progress as well as all the metrics you can think of to plot. Keep a manual control over the continue/abort decision for now.

    As a human viewing it, you'll probably have a decent prediction of whether it is worth continuing. You can then pick the metrics and limits that match your predictions best, and use them to grind the remaining tests unattended.

    Or, go meta, and set a genetic algorithm free on deciding when to stop your processing. :)

[reply]
      set a genetic algorithm free on deciding when to stop your processing:)

      I would still need to define stop criteria. In all seriousness, this is inching in that direction; but I'm trying to get a handle on how to define that heuristic.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: Instrumenting a genetic algorithm.
by kennethk (Abbot) on Aug 19, 2016 at 15:58 UTC
    I'll need to check with a colleague when they come in later today, but I thought that best practice in stopping genetic algorithms was still heuristically based. At least it used to be that they have a strong tendency toward overfitting when you only watch the cost function.

    #11929 First ask yourself `How would I do this without a computer?' Then have the computer do it the same way.

[reply]
      I thought that best practice in stopping genetic algorithms was still heuristically based

      The GA is heuristic based, it determines whether any given set of parameters results in success or failure; but there are a huge set of combinations of parameters to the GA algorithm. The 95% is a limit that decides which sets of parameters are worthy of further, exhaustive examination.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: Instrumenting a genetic algorithm.
by RichardK (Parson) on Aug 19, 2016 at 17:00 UTC

    Would calculating a rolling average over the last n results and comparing that with the total average give you a useful metric? where n is much smaller than total runs. It may give you a clue about the trend. Maybe then if the delta between the two is small it's stable/stabilizing?

    Just a thought...

[reply]

Back to Seekers of Perl Wisdom