I believe PM only counts downvotes if they are "significant" meaning they make up some non-trivial percentage of the total votes. So one downvote out of one total vote is significant, but if you then acquire a large number of upvotes, it's no longer significant.