At least one problem is that you're assigning a list into a hash — which is legal, but probably not what you want in this case:

  my %difference = grep { exists $keep_ids{ $_->{id} } } @AoH_all;
[download]

I think you want something more like this (note that I'm assigning this into an array, not a hash):

  my @keepers = grep { exists $keep_ids{ $_->{id} } } @AoH_all;
[download]