You are creating two levels of references. One when you use the anonymous array constructor. Again when you take references to arrays with only one thing apiece. You only dereference once.

What you are doing is symptomatic of something I would call, "Mental flailing". Step back. Visualize your data structure. Try to produce it. (Use Data::Dumper to see if you really are creating what you think you should be.) Then try to access that. Take it one step at a time and it will work, but you have to trust that.