Even if you put the dollar signs in, your placement of quotes in the second version would make it a symbolic reference. And in the first version it might arise still if some of the values in %hash_ref are not hashes. (That is the kind of error that can be very, very hard to track down without strict on.)

Still try this:

print keys %{ $hash_ref->{$key} };
[download]

and it will work as long as your data structure is what you think it is.

BTW for inspecting a data structure I usually use Data::Dumper.