It seems like you're structuring your data around users rather than around the pair-wise observations. Having a hashref for each user is costly. As someone else suggested, join up the user ID's into a single key. And since you only care about when both are in the same thread or both vote, you can sort them before you join them into a key. 

sub join_ids {
  return = $_[0] < $_[1] ? "$_[0];$_[1]" : "$_[1];$_[0]";
}

[download]

Depending on the nature of your user id's, you could possibly pack() them rather than doing string concatenation.

Then, for every pair observation, you're tracking two counters: number of times together and number of votes for each other. Rather than keep two separate hashes (thus duplicating the storage of keypairs) or using an HOA, why not just pack() the two numbers in the data and then unpack them and update them each time you have an observation:

# assume my %obs for holding data;

my $zerozero = pack("LL",0,0);

for (my $i=0; $i<=$#current_user_ids; $i++) {
    for (my $j=$i; $j<=$#current_user_ids; $j++) {
        my $key = join_ids( $i, $j );
        my ($seen, $voted) = unpack( "LL", $obs{ $key } || $zerozero )
+;
        $seen++;
        if ($current_votes[$i]==1 && $current_votes[$j]==1) {
            $voted++;
        }
        $obj{$key} = pack( "LL", $seen, $voted );
    }
}

[download]

That gives you a single hash, with a fairly compact data representation (even more compact if you pack the IDs). It almost exactly matches your desired output, too.

If it's still too big to manage (if the number of user-pairs is high), I'd break it up by thread count -- e.g. 1 million threads per file. The output file would be userpairs with seen counts and vote counts and then you can just sum those. (It also parallelizes across machines nicely.)

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

In reply to Re: Catching Cheaters and Saving Memory by xdg
in thread Catching Cheaters and Saving Memory by hgolden

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.