I have gathered that you have two subsets, each with about a million data points. And that you say it is very slow. How slow is "very slow"? Can you give a time frame?

I am asking because it does not appear to me that it should be very slow (unless I missed a very time-consuming step in the algorithm description). It could be that the module is doing a number of things not really necessary in your specific case and that rolling out your own sub or set of subs might be faster, or that it could be optimized some other way. But is "very slow" is a few seconds or many hours? In the latter case (assuming I have understood what needs to be done), I am convinced it could be faster. A few seconds, I would not even try to defeat the module in terms or performance. In between, well, then, it is is your and our draw. Please give a better estimate of the time frame.

Of course, any further action would require a representative dataset (perhaps significantly smaller, but sufficient for benchmarking.

@ BrowserUK: thanks for asking, I thought about more or less the same things, but somehow did not dare to ask.


In reply to Re^3: Seeking a fast sum_of_ranks_between function by Laurent_R
in thread Seeking a fast sum_of_ranks_between function by msh210

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.