And now for the ultimate speedup.

Instead of building an array of 8 million, 60-byte bitstrings and performing 32e12 xors and bitcounts, you build a single 480 MB bitstring by concatenating all the packed data.

Now, you can perform the XOR of records 0 through 7,999,999 with records 1 through 8,000,000 in one operation using substr to select the appropriate chunks of the big string. And then use substr with unpack to count the bits in each 60 byte chunk of the result.

Then you repeat the process with substrs covering 0 .. 7,999,998 & 2 .. 8,000,000. Rinse repeat till done.

8e6 XORs and 32e12 bitcounts and done.

My best guestimate is that instead of the OP code requiring upwards of 10 days on 8 cores, this'll take less than half a day on one core.

Would it be worthwhile to pre-allocate the @patNos and @data arrays with $#array = N_000_000; statements?

It wouldn't hurt, but would make little difference to the overall runtime given the scale of the processing.

Even though it would lead to 8,000,000 redundant entries giving the variance of each patent with itself (i.e., 0), might it be profitable to eliminate the next if $first == $second; statement from the inner loop of the variance calculation section? Or, avoiding redundant entries, to break the inner loop into two consecutive loops with identical BLOCK contents: for my $second (0 .. $first-1) { ... } and then for my $second ($first+1 .. $#patNos) { ... } ?

With time to give it a little more thought, there is no point in comparing both patent 1 with patent 2 & patent 2 with patent 1. With that insight, the inner loop should run from $first +1 .. $#patNos - 1. That reduces the variance calculations from 64e12 to 32e12 - 8e6 for a further halving of the runtime:

#! perl -sw
use 5.010;
use strict;

my( $i, @patNos, @data ) = 0;
$#patNos = $#data = 8e6;

while( <> ) {
    my @bits =split ',';
    $patNos[ $i ] = shift @bits;
    $data[ $i++ ] = pack 'b480', @bits;
    print "\r$.\t" unless $. % 1000;
}
say "\n", time;

open OUT, '>', 'variances.csv' or die $!;
my @variances;
for my $first ( 0 .. $#patNos-1 ) {
    for my $second ( $first+1 .. $#patNos ) {
         say OUT "$patNos[ $first ], $patNos[ $second ], ",
            unpack '%32b*', ( $data[ $first ] ^ $data[ $second ] );
    }
}
close OUT;
[download]