use a Hash of Hashes instead of a Hash of Arrays:

$ echo "host1:group1:9001:user1,user2,user3,user4,user5
host1:group2:9002:user1,user2,user3
host1:group3:9003:user1,user2,user4
host1:group4:9004:user1,user2,user5
host1:group5:9005:user1,user2
host1:group6:9006:
host2:group1:9001:user1,user2,user3,user4,user5
host2:group2:9002:user1,user2,user3
host2:group3:9003:user1,user2,user4
host2:group4:9004:user1,user2,user5
host2:group5:9005:user1,user2
host2:group1:9006:
host3:group1:9001:user1,user2,user3,user4,user5
host3:group2:9002:user1,user2,user3
host3:group3:9003:user1,user2,user4
host3:group4:9004:user1,user2,user5
host3:group5:9005:user1,user2
host3:group1:9006:" | perl -e'

use strict;
use warnings;

my %table;
while ( <> ) {
    chomp;
    my ( $host, $grp, $gid, $userid ) = split /:/;
    my $group = "$grp:$gid";
    $table{ $group }{ $userid } = 1;
    }

for my $group ( sort keys %table ) {
    print "$group:";
    my @users = sort keys %{ $table{ $group } };
    print join( ",", @users ), "\n";
    }
'
group1:9001:user1,user2,user3,user4,user5
group1:9006:
group2:9002:user1,user2,user3
group3:9003:user1,user2,user4
group4:9004:user1,user2,user5
group5:9005:user1,user2
group6:9006:
[download]

my %seen = ();
while(<FILE>) {
    chomp;
    ($host, $grp, $gid, $userid) = split(/:/, $_);
    $group = "$grp:$gid";
    # check to see if we've seen this group/user already
    next if $seen{$group}{$userid}++;
    push @{$table{$group}}, $userid;
}

Prints:
group1:9001:user1,user2,user3,user4,user5
group1:9006:
group2:9002:user1,user2,user3
group3:9003:user1,user2,user4
group4:9004:user1,user2,user5
group5:9005:user1,user2
group6:9006:
7
[download]

jwkrahn's suggestion of a hash-of-hashes is probably best, but if you're married to an anonymous array for holding your users, you can uniq-ify (and maybe sort at the same time) each user array after all data is collected.

>perl -wMstrict -le
"use List::MoreUtils qw(uniq);
 use Data::Dumper;
 ;;
 my %table = (
   foo => [ qw( c b a    b a c )   ],
   bar => [ qw( g f e d  f g d e ) ],
   baz => [ qw( i h      h i)      ],
   );
 ;;
 $_ = [ sort { $a cmp $b } uniq @$_ ] for values %table;
 ;;
 print Dumper \%table;
"
$VAR1 = {
          'bar' => [
                     'd',
                     'e',
                     'f',
                     'g'
                   ],
          'baz' => [
                     'h',
                     'i'
                   ],
          'foo' => [
                     'a',
                     'b',
                     'c'
                   ]
        };
[download]

When you need to find a unique-anything, I almost always use a hash.   In this case, the “auto-vivification” goodness of Perl is simply natural ... $myhash->{$group}{$gid}{$user}=1;.   (You don’t care what the value is.)   Any of the buckets in the entire structure that do not exist yet will, simply, appear when they are required.   Then just loop through sorted keys as you do.   To print out the final comma-separated list of users, join(',', sort keys hashref) is made for the part.

Hi again,




Thanks for all the replys, it much appreciated; I've tried all solutions (except for 'AnomalousMonk' as I work for large corporation, getting extra modules to use requires going to the DMZ which is not feasible - so I'm stuck with the bundle Solaris 10 comes with). I'm still not getting a unique list of users and this what I think is happening: When the hash pulls all the unique values ($userid) to each unique key ($group), the value is taken as a block. So for example:




This:
group1:9001:user1,user2,user3,user4,user5,user1,user2,user3,user4,user5,user1,user2,user3,user4,user5
group2:9002:user1,user2,user3,user1,user2,user3,user1,user2,user3
group3:9003:user1,user2,user4,user1,user2,user4,user1,user2,user4
group4:9004:user1,user2,user5,user1,user2,user5,user1,user2,user5
group5:9005:user1,user2,user1,user2,user1,user2
group6:9006:,,,




Is actually this in the hash:
group1:9001:"user1,user2,user3,user4,user5," "user1,user2,user3,user4,user5," "user1,user2,user3,user4,user5"
group2:9002:"user1,user2,user3," "user1,user2,user3," "user1,user2,user3"
group3:9003:"user1,user2," "user4," "user1,user2,user4," "user1,user2," "user4"
group4:9004:"user1,user2," "user5," "user1,user2," "user5," "user1,user2," "user5"
group5:9005:"user1,user2," "user1,user2," "user1,user2"
group6:9006:"," "," ","




So each block of userid's is a unique value.
I'm lost on how to do this (if there is a way).
Hope that makes sense?
Any feedback will be greatly appreciated.
Thanks again.

As another monk mentioned, most of the reasons people think they can't install from CPAN are in fact non-reasons.

But either way, the uniq function from List::MoreUtils is easy for you to implement yourself:

sub uniq (@) {
    my %seen = ();
    grep { not $seen{$_}++ } @_;
}
[download]

As it happens, I just copied and pasted that code from List/MoreUtils.pm. However, one reason to prefer List::MoreUtils over a home-made version, is that List::MoreUtils provides an XS version which (over large lists) may run a little faster.

In the meantime, Yes, even you can use CPAN