Yep. That's the essence of the winning-without-fighting approach I mentioned. If you frequently had a couple of addresses to whitelist, instead of only very occasionally, you'd sort the probable-spam folder (which I called greybox) by spam score, and then you'd have exactly that approach.