You're welcome :) But you shouldn't just give up.

Just because your current methods are intractable, it doesn't mean that there isn't a viable approach to solving your underlying quest. It just means that you should look at different approaches to that problem.

For example, the way you are currently storing your data--as tab-delimited streams of 1s and 2s is hugely wasteful of both storage and processing time. Your described 1e6 x 1e6 dataset file must be something close to 2TB (2e12) bytes of data.

When you realise that the 1s & 2s can be represented by 0s & 1s, and can therefore be stored binary encoded, the size of your data file could be reduced to something like 116 GB. That's the same information, but less than 6% of the volume of data to read, store and manipulate.

Or if, as your other thread suggests, that you're looking for positions where there is a minimum overlap of ten datapoints then that is only 1024 possible search strings. So, in a single pass, you could index all your records by the 1024 10-value substrings they contain. That might take 1/2 a day or so.

Now, with the index, you can limit the number of records you need to search. Assuming a random distribution, that means you've reduced your problem by 3 orders of magnitude. Suddenly, 72 years becomes more like 25 days. Which is a whole heap more tractable!

Of course, there are no guarantees at this stage that such a reduction is achievable, since we are still in the dark as the the reality of your actual problem. And unless you describe it, you will not be able to gain the benefit of the many clever minds here, considering what other methods might be fruitful.