"\x61" interpolates down to the letter "a". The value of 97 stringifies to the string, "97", and no further interpolation is done. Therefore, you're storing a hash key named 'a', and looking for a hash key named '97'.

The fact that the literal "\x61" interpolates to the character 'a' is discussed in perlop under Quote-like Operators. If you want the literal decimal number 97 to be used as the ASCII (or code-point) value of a character, you should use chr: print $hash{chr(97)}, "\n";