NaN is very much not zero

More than that - in the world of numbers, NaN is very much not anything ... and I can't think of anything falser than "not anything".

I assume you aren't making any of the "Inf"s "false"

That's correct - an Inf is just like any other non-zero number, except bigger. No doubt in my mind that Inf should be true.

when I test a number for "truth", I usually think of it as short-hand for comparing a number to 0

You're probably not the only one who does that - yet I've created a situation where that's not what "truth" is testing.
That provides an opportunity for confusion and misunderstanding to arise, which is definitely not such a good thing.

Cheers,
Rob