jethro:

I like that approach, but rather than excluding columns one-by-one, I'd include them one-by-one, like this:

$ cat 896650.pl
#!/usr/bin/perl
use strict;
use warnings;

# Generate values
my @cols = ( 0 .. 10 );
my $rows=50000;
my @data = ( [ map { int 100*rand } @cols ] );
my @gen = ( map { [ 100*rand, int 100*(1+$_*$_*$_*rand) ] } @cols );
for my $r (1 .. $rows) {
    for my $c ( @cols ) {
        $data[$r][$c] = int( $data[$r-1][$c]
            + $gen[$c][0]*rand) % ($gen[$c][1]);
    }
}

# Count distinct column values
my %H;
for (@data) {
    my @v=@$_;
    for my $c (@cols) {
        $H{$c}{$v[$c]}++;
    }
}

# Sort so most diverse comes first
my @colrank = sort { $$b[1] <=> $$a[1] }
              map { [ $_, scalar keys %{$H{$_}} ] } @cols;
my @colOrder = map { $$_[0] } @colrank;

for my $r (@colrank) {
    print "col $$r[0] has $$r[1] values\n";
}
print join(", ", @colOrder), "\n";

print scalar(@data), " rows to process.\n";

my @tmpdata = @data;
my @keycols = ( shift @colOrder );
while (keys %H < @tmpdata) {
    print scalar(@tmpdata), " rows to process.\n";
    %H=();
    for (@tmpdata) {
        my $key=join(".", @$_[@keycols]);
        $H{$key}++;
    }
    print "cols(", join(",",@keycols), ") give ",
          scalar(keys %H), " rows.\n";
    push @keycols, shift @colOrder;
}

$ perl 896650.pl
col 10 has 37754 values
col 9 has 29806 values
col 8 has 29772 values
col 7 has 22835 values
col 6 has 16081 values
col 5 has 10136 values
col 3 has 2726 values
col 4 has 559 values
col 2 has 216 values
col 1 has 176 values
col 0 has 1 values
10, 9, 8, 7, 6, 5, 3, 4, 2, 1, 0
50001 rows to process.
50001 rows to process.
cols(10) give 37754 rows.
50001 rows to process.
cols(10,9) give 49996 rows.
50001 rows to process.
cols(10,9,8) give 50001 rows.

[download]

However, I'd wonder what the utility of it would be...

I was thinking that one possible improvement of my code would be to remove the "duplicate" rows after determining each key, then redo the column count so you could get the best remaining key from the list. It certainly wouldn't be the optimal solution, but I'd bet it gets pretty close.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

In reply to Re^2: Finding Keys from data by roboticus
in thread Finding Keys from data by aartist

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.