in reply to Eliminate exact duplicates from array of hashes

Thank you for the feedback (and for confirming there is not a standard/build-in solution to do this). My @test_data should contain only hashes as in my example (and probably with a bit of cleaning in the pre-process) always with the two (identical) keys. The values of the hashes should any digit/literal (words and so on), possibly containing also characters such as ,|'". (unicode strings). The size of @test_data can be big (100.000 hashes), but performance (time) is not an issue in this context.

Replies are listed 'Best First'.
Re^2: Eliminate exact duplicates from array of hashes
by huck (Prior) on Oct 09, 2019 at 18:44 UTC

    Now knowing more about the problem i might try 

    use strict; use warnings; 
use Data::Dumper;

my @test_data = (
    { Tag1 => "1", Tag2 => "a" },
    { Tag1 => "1", Tag2 => "a" },
    { Tag1 => "1", Tag2 => "b" },
    { Tag1 => "1", Tag2 => "c" },
    { Tag1 => "1", Tag2 => "a" },
    { Tag1 => "2", Tag2 => "a" },
    { Tag1 => "2", Tag2 => "d" },
    { Tag1 => "2", Tag2 => "a" },
    { Tag1 => "3"},
    { Tag1 => "sun", Tag2 => "a" },
    { Tag1 => "sun", Tag2 => "a" },
);
my %found;
my @unique;  
for my $grp (@test_data) { 
  my $t1=$grp->{Tag1}//''; 
  my $t2=$grp->{Tag2}//''; 
  next if ($found{$t1}{$t2}); 
  push @unique,$grp; 
  $found{$t1}{$t2}=1; 
  } 

print Dumper \@unique;

    [download]
    as it saves the cost of serializing and the cost of repeated storage of the characters Tag1/Tag2 at the expense of only checking the two keys AND representing the undefined key value as a zero length character scalar. 

    $VAR1 = [
          {
            'Tag1' => '1',
            'Tag2' => 'a'
          },
          {
            'Tag2' => 'b',
            'Tag1' => '1'
          },
          {
            'Tag1' => '1',
            'Tag2' => 'c'
          },
          {
            'Tag1' => '2',
            'Tag2' => 'a'
          },
          {
            'Tag2' => 'd',
            'Tag1' => '2'
          },
          {
            'Tag1' => '3'
          },
          {
            'Tag1' => 'sun',
            'Tag2' => 'a'
          }
        ];

    [download]

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom