Not to add to your paranoia ... but that's not necessarily true. There's some monks I respect greatly and if they were the only downvote, I would go back and rethink.

So my advice, as many others, is to not worry about your node rep and just worry about your rep. If a fellow monk takes the time to explain a downvote, then rethink. If a fellow monk does not explain his downvote then don't worry about it.