You're right. Sorry for the assumption.

I think I see the cause of the problem. If you look at pp_add (pp_hot.c) you'll see that it checks for the IV and UV flags on the scalar (SvIOK, SvUOK) and since neither of these is set it falls through to do the arithmetic with NVs.

To fix this the magic code would need to be hoisted out of sv_2iv_flags() and the check for SvIOK (or SvUOK) done on the scalar returned by the numeric overload.

Fixing this for every operator would be a big change.

Edit: pp_add is in pp_hot.c, not sv.c