I think we have a XY Problem problem here

To store 10^12 arrays of size 1000 you would need at least 1000 Terabyte. Since you probably don't have that amount of storage I assume you already multiplied your 1 million arrays to arrive at that number. Whatever you do you will never speed up the comparisions enough to make 10^12 comparisions fast. You have to ask instead how to avoid making 10^12 comparisions.

I also assume the sets of different elements is relatively small and the criteria are not fixed, i.e. it is not always element 3 you are comparing.

Then a solution could be to use a database or even simple files where you store the id of all arrays that have element m as the n-th element

So if you use files, you would in a first step preprocess your arrays and create the files a0,a1,a2...a1000,b1...b1000,c1...c1000,... . In file a0 you would store all array-ids where an a is in position 0. So both of your example arrays above would be listed in a0.

That would be a few thousand files, but a modern filesystem can handle that easily. Now your example search would work differently: You would simply list all array id-pairs in the files a0+b0,a0+c0,a0+d0...,b0+c0,b0+d0,... except for the files a0+x0,b0+x0... If there are none, resume with a1+b1,a1+c1...

Note that if you have more criteria to further restrict your solution space, those criteria should be able to prune down the file pairs *before* you have to read any of these files

This solution might not be the best way to represent your data. Provide more information and a better solution might surface

In reply to Re: speedy way to compare elements in pairs of arrays? by jethro
in thread speedy way to compare elements in pairs of arrays? by CaptainF

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.