jethro:

I like that approach, but rather than excluding columns one-by-one, I'd include them one-by-one, like this:

$ cat 896650.pl
#!/usr/bin/perl
use strict;
use warnings;

# Generate values
my @cols = ( 0 .. 10 );
my $rows=50000;
my @data = ( [ map { int 100*rand } @cols ] );
my @gen = ( map { [ 100*rand, int 100*(1+$_*$_*$_*rand) ] } @cols );
for my $r (1 .. $rows) {
    for my $c ( @cols ) {
        $data[$r][$c] = int( $data[$r-1][$c]
            + $gen[$c][0]*rand) % ($gen[$c][1]);
    }
}

# Count distinct column values
my %H;
for (@data) {
    my @v=@$_;
    for my $c (@cols) {
        $H{$c}{$v[$c]}++;
    }
}

# Sort so most diverse comes first
my @colrank = sort { $$b[1] <=> $$a[1] }
              map { [ $_, scalar keys %{$H{$_}} ] } @cols;
my @colOrder = map { $$_[0] } @colrank;

for my $r (@colrank) {
    print "col $$r[0] has $$r[1] values\n";
}
print join(", ", @colOrder), "\n";

print scalar(@data), " rows to process.\n";

my @tmpdata = @data;
my @keycols = ( shift @colOrder );
while (keys %H < @tmpdata) {
    print scalar(@tmpdata), " rows to process.\n";
    %H=();
    for (@tmpdata) {
        my $key=join(".", @$_[@keycols]);
        $H{$key}++;
    }
    print "cols(", join(",",@keycols), ") give ",
          scalar(keys %H), " rows.\n";
    push @keycols, shift @colOrder;
}

$ perl 896650.pl
col 10 has 37754 values
col 9 has 29806 values
col 8 has 29772 values
col 7 has 22835 values
col 6 has 16081 values
col 5 has 10136 values
col 3 has 2726 values
col 4 has 559 values
col 2 has 216 values
col 1 has 176 values
col 0 has 1 values
10, 9, 8, 7, 6, 5, 3, 4, 2, 1, 0
50001 rows to process.
50001 rows to process.
cols(10) give 37754 rows.
50001 rows to process.
cols(10,9) give 49996 rows.
50001 rows to process.
cols(10,9,8) give 50001 rows.
[download]

However, I'd wonder what the utility of it would be...

I was thinking that one possible improvement of my code would be to remove the "duplicate" rows after determining each key, then redo the column count so you could get the best remaining key from the list. It certainly wouldn't be the optimal solution, but I'd bet it gets pretty close.

...roboticus

When your only tool is a hammer, all problems look like your thumb.