Why are hex values not interpreted as numbers

They are, if they appear as such in your code, e.g.:

perl -le 'print 0x41'
65
[download]

They are not, if the hexadecimal notation happens to be a string. The string "0x41" in numerical context yields 0, since from left to right, the numerical part ends after 0.

perl -le 'print 0 + "0x41"'
0
[download]

Same for strings with leading decimal numbers:

perl -le 'print 0+"3.1415pi--roughly"'
3.1415
[download]

But perl doesn't look at your string trying to guess whether it is a valid hex number. If you want a string to be interpreted as a hex number, use hex. It will happily remove the leading "0x":

perl -le 'print hex "0x41"'
65
[download]