I think lzcd is right. I would guess it's just a concidence that you haven't noticed XP changes after using up all your votes for a day.

If you think back to when you were an intitiate, you'll probably remember seeing XP changes before you even had votes!