Comparing two general hierarchical data structures is in general a hard problem. First, you have to establish a criterion for equivalency. Do hash values have to be exactly the same, or is it the values' contents? Must arrays have exactly the same elements in the same order, or is it that they form equivalent sets good enough? Second, you have to come up with a search strategy.

For instance, for a hash of hashes and assuming $retrieved is a superset of $given, the following can be used:

my $result = {};
foreach my $main_key (keys %$retrieved) {
   unless (exists $given->{$main_key} ) {
      $result->{$main_key} = $retrieved->{$main_key};
      next;
   }
   # The key exists, compare subhashes
   foreach my $sub_key (keys %$main_key) {
      $result->{$main_key}{$sub_key} = $retrieved->{$main_key}{$sub_ke
+y}
         unless exists $given->{$main_key}{$sub_key}
            && $given->{$main_key}{$sub_key} eq $retrieved->{$main_key
+}{$sub_key};
   }
}
[download]

The idea is that given your data structure and equivalence criteria, you can drill down and simply do comparisons, rather than deletions. This should be quicker. For your particular application, I cannot discern your equivalence criterion, so I'll stop here.