After some more reading I found that my first assumption is only half true.
$1-$9 are ALWAYS backrefrences, where $10 and up are only backrefrences if more than 10 parens were used.
This mixes with the comments about a variable that starts with a digit, can only have digits.

So $10 is taken as the whole name of a variable (undefined)
$1_ gets treated like $1 because _ is not a digit.

Thanks for all the help, I'm slowly understanding the subtle nature of Perl :)