Dru has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

What is the best way to remove duplicates from an array of arrays?

Here's how I'm printing it:
# Print out the array of arrays for my $x (0 .. $#AoA){ print "$AoA[$x][0] -> $AoA[$x][1]\n"; } Output: 192.168.19.18 -> 10.151.150.29.2703: 192.168.19.1 -> 10.151.150.33.2703: 192.168.19.18 -> 10.151.150.29.2703: 192.168.19.1 -> 10.151.150.33.2703: 192.168.19.1 -> 10.2.252.10.53: 192.168.19.1 -> 10.2.252.10.53: 192.168.19.1 -> 10.6.60.197.113: 192.168.19.18 -> 10.151.150.29.2703: 192.168.19.18 -> 10.151.150.29.2703: 192.168.19.18 -> 10.151.150.29.2703: 192.168.19.18 -> 10.151.150.29.2703: 192.168.19.18 -> 10.151.150.29.2703:
Thanks,
Dru

Replies are listed 'Best First'.
Re: Remove Duplicates from an Array of Arrays
by davorg (Chancellor) on Jul 14, 2004 at 15:22 UTC
    my %seen; for (0 .. $#AoA) { print "$AoA[$_][0] -> $AoA[$_][1]\n" unless $seen{$AoA[$_][0]}{$AoA[$_][1]}++; }

    (although that doesn't remove the duplicates, it just stops you from printing them)

    --
    <http://www.dave.org.uk>

    "The first rule of Perl club is you do not talk about Perl club."
    -- Chip Salzenberg

Re: Remove Duplicates from an Array of Arrays
by ccn (Vicar) on Jul 14, 2004 at 15:27 UTC
    my %H = map {("$_->[0]!$_->[1]" => 1)} @AoA; @AoA = map{[split /!/]} keys %H;
Re: Remove Duplicates from an Array of Arrays
by dragonchild (Archbishop) on Jul 14, 2004 at 15:31 UTC
    When removing duplicates, the standard way is to use a hash. The key of the hash is what has to be unique. A little manipulation shoudl get you what you need.

    ------
    We are the carpenters and bricklayers of the Information Age.

    Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

    I shouldn't have to say this, but any code, unless otherwise stated, is untested

Re: Remove Duplicates from an Array of Arrays
by rjbs (Pilgrim) on Jul 14, 2004 at 15:35 UTC
    Consider mapping?
    my %seen_values; for my $aref (@array_of_arrayrefs) { $aref = [ map { $seen_values{$_}++ ? () : $_ } @$aref ]; }
    For every arrayref in the list, transform the arrayref's array as follows: if the value has been seen before, ignore it. Otherwise, include it. Either way, make sure you note that you've seen it.

    NOTE! This removes any value that's seen more than once. I think, though, that you want to remove entire pairs that are seen more than once. Your use of arrows makes it seem like you care about pairs. After all, it's like =>, used in marking hash pairs. So, if you indeed meant duplicated arrayrefs, not duplicated deep values...
    my %seen_values; my @new_AofA = map { $seen_values{$_->[0]}{$_->[1]}++ ? () : $_ } @Aof +A;
    It's the same thing, but we're using a two-level hash. After all, it's a lot like what you were thinking, isn't it?

    If this isn't clear, I can elaborate on how it works. I know using $_ makes some people glaze over...
    rjbs
      Doesn't it seem like grep is a more natural choice?
      my %seen_values; my @new_AofA = grep { !($seen_values{$_->[0]}{$_->[1]}++) } @AofA;

      We're not really tightening our belts, it just feels that way because we're getting fatter.
        Yes, you're quite right. I think I got map in my head for some reason and ran with it! Anyway, fortunately both will work, and the use of %seen is the thing Dru probably needs. Roy's grep example helps make that clearer.
        rjbs
        Given the data, using old-style multidimensional hashing would work also:
        my @new_AofA = grep { !$seen{$_->[0],$_->[1]}++ } @AofA;
Re: Remove Duplicates from an Array of Arrays
by NetWallah (Canon) on Jul 14, 2004 at 20:33 UTC
    This looks like a "netstat" or NAT table.

    If so, a simple hash may be a better structure to hold the information. You can transform the AOA into a hash while eliminating duplicates using this simple code:

    my @AoA=([11,1],[22,2],[11,1],[44,4]); my %h; $h{$_->[0]} = $_->[1] for @AoA; print qq($_ -> $h{$_}\n) for sort keys %h --Output--- 11 -> 1 22 -> 2 44 -> 4

        Earth first! (We'll rob the other planets later)

Re: Remove Duplicates from an Array of Arrays
by Dru (Hermit) on Jul 14, 2004 at 18:24 UTC
    Monks,

    Thanks, lots of good solutions here. I'll play with a few and pick my favorite.