The reason is that with numbers you can do this:

my $a = 1;
my $b = $a;
$a++;
$b++;
say $a; # says "2"
say $b; # says "2"
[download]

But if $a and $b are overloaded references to numbers, then they refer to the same object, so you've actually incremented the same object twice. $a and $b are both 3.

Perl forces you to deal with (or at least acknowledge that you've thought about) this problem by in certain circumstances forcing you to provide an overload for "=".