Just adding this on- the reason I like the above solution so much is that it penalizes low confidence entries, but not with any kind of magic number- it does with it perhaps the most time tested and mathematically backed up method of penalizing low confidence entries ever! | [reply] |
While I agree, with the use of Margin of Error/Confidence values, I think your approach is overly pessimistic. Take for example an entry that gets a single minimum vote (1 or 0) out of 5. This is just as likely to be deceptive and your approach would, incorrectly I believe, drive this value farther below minimum.
As this problem is most noticeable only at/near 100% & 0%, I suggest another approach. Generate 2 values, Score + Error and Score - Error. Trim these values to the range in question, and average the Trimmed values for a final score. This would have the effect of moving scores that are pegged at the end of the range inward by Half their *possible* error. Scores near the center of the range will be left largely unaffected. | [reply] |
I agree that the approach is probably too pessimistic. It essentially takes the view that all uncertainty is downside.
I would be reluctant to fudge the calculation of the mean. Which aspect of the data has more "information" to it? The mean or the confidence? That should guide you about how to combine the two.
To my eye the least invasive treatment would be to compute the means and the standard errors, and then use the standard error to break ties when sorting by the means. Obviously you need the actual values in the population for this (to compute the standard deviation). For the other ones, maybe you can stick in a "default" standard error (don't know what you're scoring, so who knows).
| [reply] |