Why do you want to go through so much trouble to avoid putting the output in a similarly configured hash?  $output{xx}[1] seems like a small price to pay to avoid using the symbolic references you seem to be talking yourself into.