But why doesn't this confuse Perl?

Using the output of your first snippet as an example,

SV = PVIV(0x8150b10) at 0x814f6b4   <-- Can contain an IV and PV
  REFCNT = 1
  FLAGS = (IOK,pIOK)                <-- Only contains an IV
  IV = 123
  PV = 0x81623f0 "abc"\0
  CUR = 3
  LEN = 4
SV = PVIV(0x8150b10) at 0x814f6b4   <-- Can contain an IV and PV
  REFCNT = 1
  FLAGS = (POK,pPOK)                <-- Only contains a PV
  IV = 123
  PV = 0x81623f0 "123"\0
  CUR = 3
  LEN = 4
[download]

The type of the scalar is promoted as necessary. Downgrading would be a waste of time, so flags are used to indicate what kind of value the scalar contains.

Note that it's possible to have a scalar that has both an IV and PV.

$ perl -MDevel::Peek -e'$x = 123; Dump $x; "$x"; Dump $x'
SV = IV(0x816a41c) at 0x814f69c
  REFCNT = 1
  FLAGS = (IOK,pIOK)                <-- Only contains an IV
  IV = 123
SV = PVIV(0x8150b10) at 0x814f69c
  REFCNT = 1
  FLAGS = (IOK,POK,pIOK,pPOK)       <-- Stringification has
  IV = 123                              been cached.
  PV = 0x81651a0 "123"\0
  CUR = 3
  LEN = 4
[download]

On the other hand, Storable seems to blindly favor PV when both IV and PV exist,

When both IV and PV exist (i.e. when the scalar is a PVIV or subtype), or when POK is set? The former would definitely be a bug. Otherwise, read on.

If a scalar contains both a number and a string (POK and either IOK or NOK), the situation is intended to be one of the following (and almost always is):

• The string is a stringification of the IV.
• The string is a stringification of the NV.
• The number is a numification of the string.

If we accept that more general dualvars can't be stored, we should be able to store only one of the string or the number. So which one should we used when?

The string is a stringification of the IV.	Stringification of IVs is lossless, and so is their numification.	Either the string or the IV will do.
The string is a stringification of the NV.	Stringification of NVs is practically lossless, and so is their numification.	Either the string or the NV will do.
The number is a numification of the string.	Numification of non-numeric strings is lossy. e.g 0+"abc" ne "abc". Numification of partially numeric strings is lossy. e.g 0+"123abc" ne "123abc". Numification of some numeric strings is lossy. e.g 0+"0.50" ne "0.50".	Necessarily, we need to store the string.

As such, always storing the string is sufficient and appropriate.

Update: Storing the string instead of the number also allows us to store '0 but true' and '0E0', true values that evaluate to zero.