No. I believe a vote is a message from me to the author of the node. It is a means for me to say - " Yes - I agree" or "Ah, I hadn't looked at it that way". Accumulated reputation shows the trend of feeling by the users of the site about a particular node. If the author cannot see the reputation accrued then it is a wasted vote. In my opinion.

The XP issue is entirely separate and though nice to have at the end of the day if you frequent the site over a long enough period you will reach sainthood so in my book that is the least of the issues to focus on.