Should be relatively easy to code yourself...

sub object_set {
  my($lista,$listb) = @_;
  my(%seen,%seen2);

  for(@$lista) {
    push @{$seen{$_->id}}, $_;
  }
  for(@$listb) {
    if($seen{$_->id}) {
      push @{$seen2{$_->id}}, @{delete($seen{$_->id})}, $_;
    } elsif($seen2{$_->id}) {
      push @{$seen2{$_->id}}, $_;
    } else {
      push @{$seen{$_->id}}, $_;
    }
  }
  return { diff => \%seen, int => \%seen2 };
}
[download]

Not tested, but the idea is there and easily tweaked. Since you don't want to use refaddrs I assume you can have multiple references that have the same id, so I pushed them onto an array to catch them all. Hope that helps.