The reason why the same offset is seen in strings with a match is because these are biological data.

Are you saying that the substrings you are interested in will always appear at the same offset in both strings?

This is important, because if it is the case, your entire dataset can be processed in under 90 seconds.

P:\test>484593-9 bioman.dat 000:001 L[072] (0072,0072)'TTCAGAGACACAGAACCCTGCTCCGACTATTGCCTCTCTCACATCATCA +A' 000:002 L[1271] (1271,1271)'TGCGGAACTCCTTGCAGCTTGTTTCGCTCGCAGCCGGTCTGGAGCGAAA +C' 000:003 L[225] (0225,0225)'AATCCTCACAATACCACAGAGTCTAGACTCGTGGTGGACTTCTCTCAAT +T' 000:004 L[191] (0191,0191)'TCGTGTTACAGGCGGTGTGTTTCTTGTTGACAAAAATCCTCACAATACC +A' 000:005 L[269] (0269,0269)'TCAATTTTCTAGGGGGAACACCCGGGTGTCCTGGCCAAAATTCGCAGTC +C' 001:002 L[072] (0072,0072)'TTCCGGAACAGTAAACCCTGTTCCGACTACTGCCTCACCCATATCGTCA +A' 001:003 L[072] (0072,0072)'TTCCGGAACAGTAAACCCTGTTCCGACTACTGCCTCACCCATATCGTCA +A' 001:004 L[080] (0080,0080)'CAGTAAACCCTGTTCCGACTACTGCCTCACCCATATCGTCAATCTTCTC +G' 001:005 L[072] (0072,0072)'TTCCGGAACAGTAAACCCTGTTCCGACTACTGCCTCACCCATATCGTCA +A' 002:003 L[320] (0320,0320)'CAACCTCCAATCACTTACCAACCTCCTGTCCTCCAACTTGTCCTGGCTA +T' 002:004 L[191] (0191,0191)'TCGTGTTACAGGCGGTGTGTTTCTTGTTGACAAAAATCCTCACAATACC +A' 002:005 L[269] (0269,0269)'TCAATTTTCTAGGGGGAACACCCGGGTGTCCTGGCCAAAATTCGCAGTC +C' 003:004 L[161] (0161,0161)'ACATCACATCAGGACTCCTAGGACCCCTGCTCGTGTTACAGGCGGTGTG +T' 003:005 L[510] (0510,0510)'ACCATGCAAAACCTGCACAACTCTTGCTCAAGGAACCTCTATGTTTCCC +T' 004:005 L[148] (0148,0148)'ATGAACATGGAGAACATCACATCAGGACTCCTAGGACCCCTGCTCGTGT +T' 15 trials of bioman.dat ( 27.018ms total), 1.801ms/trial

That projects to a total time for your 300x3k strings of 80.78 seconds!

There is no trick to that result. Very simple pure perl code that just makes use of the knowledge that the sequences of interest always appear at the same offsets in both strings.


If that is not the case, then a much faster solution will be possible if you can guarentee that all equal-length matching sequences within a given pair will be in the same relative order, even if at different offsets?

To clarify that. Say that there are three equal-length, matching strings (xxx, yyy & zzz) in a given pair of sequences (A & B). If they will always appear such that offset(xxx) < offset(yyy) < offset(zzz) in both A & B. Eg.

A: ....xxx...yyy...zzz B: xxx....yyy.......zzz.... or A: .xxx........yyy...zzz....... b: .............xxx.yyy.zzz etc.

Then a much faster solution is possible than the general case where the matching strings might appear in any order.

A: ...xxx...yyy...zzz B: yyy...xxx...zzz...

Likewise, if equal-length matching sequences would appear with the same ordering and the same relative offsets:

A: xxx...yyy...zzz........... B: .....xxx...yyy...zzz......

That knowledge would also speed up the search considerably.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

In reply to Re^7: Search for identical substrings by BrowserUk
in thread Search for identical substrings by bioMan

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.