The former suggestion sounds OK to me, but I think the latter would do more harm than good. An unpopular upvote/downvote != an indiscriminate one. Having people who don't want to lose XP worrying about whether their votes go along with the majority strikes me as a worse situation than having people who so want to gain XP that they're voting indiscriminately doing so.