See my point? There's no possible way to ensure that only skilled down-voters are counted, so not counting any downvotes at all actually sovles the problem.

There is no problem. There's no need to discern between skilled and not so skilled votes either way, since most of the regulars on this site are skilled people, in more than perl.