I wonder why the initial developers of this voting system didn't separate votes you get from other users voting for your posts, and your own points you get while voting for other posts. It seems to me that these two numbers are very different in their nature. The former is the quality of the material you write (and which could be, eventually, considered by third parties interested in one's abilities) while the latter means that you can recognize good post, but it doesn't prove your ability to write similar (or better) one.
So, if anybody in the future seriously considers to change the XP or the voting system, I think this is the first direction to be examined.