Once the reference is assigned a location, it never moves. The data behind it may get moved, but the reference you see doesn't.
There are only four major gotchas to using references as hash keys that I know of.
It depends on stringification, so if the generator of the reference likes overloading, all bets are off.
From the string representation of a reference you can't get the reference back.
Names are guaranteed unique at any point in time, but not over time. As demonstrated in my code sample, it is possible for 2 references to get the same name if the second is created after the first is destroyed.
If you ever make your program persistent by freezing and thawing data structures (eg with Storable or Data::Dumper) then the reconstructed references will be at new addresses.