while i don't have any immediate insight, if you haven't already compared Array::AsHash to Tie::RefHash ( or Tie::RefHash::Nestable ) those modules might offer some other insight.

those modules do something similar ( allow refs as hash keys ).