Displaying a users XP to the user is of particular importance because it describes the relationship they have achieved with the community, albeit in a simple one-dimensional numeric. If you had -50 XP, you are receiving a pretty strong signal. Likewise, if you had 5000 XP, you must be doing something right.

With respect to showing your XP for all to see, it is not strictly necessary, but since most would likely want to see how they are progressing with respect to other community members, had it not been visible historically, there would certainly be a movement to make these numbers visible to all. Competition for higher XP is, in this regard, healthy.

XP is not specifically a goal, unless you make it one. Perhaps someone could somehow "engineer" their posts to garner a lot of XP. This would likely involve helping out the community a lot. Is answering more questions to gain more XP a bad thing?

Don't forget, though, that you can't make votes appear out of thin air. Real people actually have to vote, so there is a moderating mechanism there. If people feel like they are being cheated somehow, or that that user is trying a little too hard, they can always use their votes in the negative capacity to discourage it.