...the voter gains XP and advances in levels from upvotes...

Not only from upvotes...the voting node says: "You have 25% chance of gaining 1 XP every time you vote on a node. If, however, you cast more downvotes than upvotes, you can be penalized by losing XP".

So you get XP for both up votes and down votes. Unless you downvote more. Surely there are only two reasons why that would happen...either you are being mischievous (just the kind of behaviour the OP was concerned about) or you don't get any value from the content and will leave naturally anyway.