the reason for the two hashes is because I wanted to simplify the code

You didn't do that. Your data structure is horribly complex. What's the point of having a hash for every key! (And don't say "1" and "2" could be the same string because then your goal makes no sense at all. There wouldn't be a right "hm" to fetch.)

Is there a performance gained by using an array over the hash?

Performance? Not really. The problem is that emulating a hash using an array is one more thing that can go wrong compared to just using an array.