Have a glance at Voting/experience system and the "How can I gain/lose XP?" subsection there.
There's more than one way to get XP...

Cheers, Sören