sauoq++ has already made the point, but I wanted to reinforce the point about this:

by having more users on the site, doesn't that "bloat" my XP in comparison to the XP required to go up levels?

When there are fewer people, it also means votes are concentrated on fewer people.

The key issue here is the votes/day limit. More active users who vote means also roughly as many more new posts/day that the ratio remains close to equal.

Makeshifts last the longest.