As a reader you typically don't know what the vote totals are and even with the vote totals they won't tell you anything about the quality or accuracy of the comment. I have not heard an idea about this "problem" which won't be a royal pain in the general case. I don't want to have to send people private messages when I vote. I don't want to have to write public messages when I vote. I don't want my access to positive or negative votes restricted because I'm not interesting in justifying my every action.

Generally I vote because I liked/disliked something, agreed/disagreed with, am annoyed with the person, am annoyed with the thread, am being deliberately contrary to someone's expectations (this is usually a -- vote for people who complain about -- voting), or most commonly am just exerting some control over the sorting of the nodes in the thread.