From what you show us, this interpretation sounds correct. Maybe you want to take a closer look at what's currently in %labels by using Data::Dumper:

print Dumper \%labels;
[download]

Also, if you have use strict; at the top of your program (or in the relevant block), that will prevent such bad things as using a string where a reference is expected. If you don't have it, now is a good time to start with the innermost block you're changing and putting use strict; there, just to catch such things.