I agree... that would probably eliminate the possibility of abuse. Of course, it would probably also ensure that it was almost never used. (I have a feeling most people would prefer to lose a vote than an XP.¹) So, all the work to implement it would be for nought... :-)

1. Those with higher XP tend to be less concerned about it but they also have more votes and tend to be less concerned about wasting them as well. Whereas, those with less votes also have less XP and are still trying to gain levels. I'm sure there would be exceptions, mostly based on altruism. For instance, if I downvoted someone by accident, I might like that feature just to take back my down vote, but I wouldn't use it just to get back a vote wasted by accident if I didn't feel bad about the result. Besides, I don't think downvoting by accident is a frequent occurence...

-sauoq
"My two cents aren't worth a dime.";