While I agree, with the use of Margin of Error/Confidence values, I think your approach is overly pessimistic. Take for example an entry that gets a single minimum vote (1 or 0) out of 5. This is just as likely to be deceptive and your approach would, incorrectly I believe, drive this value farther below minimum.
As this problem is most noticeable only at/near 100% & 0%, I suggest another approach. Generate 2 values, Score + Error and Score - Error. Trim these values to the range in question, and average the Trimmed values for a final score. This would have the effect of moving scores that are pegged at the end of the range inward by Half their *possible* error. Scores near the center of the range will be left largely unaffected.