Hashes and array elements are scalar values so can only hold references to nested hashes or arrays. Therefore hash_of_hash_of_arrays{$outerkey} is a reference to a hash. To get the hash you need for keys you need to dereference it using %{}: %{$hash_of_hash_of_arrays{$outerkey}}.

Note that $, % and @ don't tell you what type variables are, they tell you what type of thing an expression returns. So $ref returns a scalar which may be a reference to an array or a hash. @$ref and @{$ref} return arrays (they are equivalent) by dereferencing $ref. The {} version is much clearer (and often required) when the expression used to get the reference is complicated.