I have not yet downvoted anyone. Despite reading a few questions I considered lame, nothing has been that bad to me. I will upvote a question that gets a good response with informative answers or an individual answer I like.
However I will often not vote for an obviously well-thoughtout answer because that answer is over my head. If I don't understand an answer I don't feel qualified to vote on it.
By the way, what is the maximum number of votes an individual can have. Do Saints spend all day voting?