That may matter for new users, but for monks like me who already have lots of XP, I don't care about that little penalty much. Plus, as I also have lots of votes, I can usually burn away the penalty by upvoting random nodes. Thus, I downvote lots of nodes I don't like.