First -- I a think that showing the aggregate rather than the +/- detail is a good thing. If you are writing good notes, then the occasional (usually accidental) down vote is not a problem.

But, I do have an issue with the aggregate system: Occasionally I will up vote a new node and the score that gets displayed is zero. That means that before my +1 the score was -1. I do not consider 50% of the votes an "insignificant number".