shamshersingh has asked for the wisdom of the Perl Monks concerning the following question:

I have a large set (100000+) of short DNA reads 20 characters long. I need to compares all reads against each other and pull out those that vary by just 1 position. Heres the script I came up with. 
$| = 1;
my $compare_count = 0;
for (my $i = 0; $i < @kmers; $i++ ) {
    for (my $j = $i + 1; $j < @kmers; $j++ ) {
        print "\rComparing sequence $i to $j";
        my @result = PCCompare::dissimilarity($kmers[$i], $kmers[$j], 
+1);
        if ($result[0] == 1) {
            print "\rMatch found: $kmers[$i], $kmers[$j]\n";
            push @variant_kmers, ($kmers[$i], $kmers[$j]);
        }
        $compare_count++;
    }
}
print "\rFinished: $compare_count comparisions made.\n";

[download]
The problem is that this loop runs very very slow. It takes on the orders of days to process 100000 sequences. Is there a way to make the process faster?

Replies are listed 'Best First'.
Re: Comparing a large set of DNA sequences
by BrowserUk (Patriarch) on Nov 09, 2011 at 23:09 UTC

    This will perform the same job, but run significantly faster (Approx 7 hours). 

    $| = 1;
my $compare_count = 0;
for (my $i = 0; $i < @kmers; $i++ ) {
    for (my $j = $i + 1; $j < @kmers; $j++ ) {
        my $sims = ( $kmers[ $i ] ^ $kmers[ $j ] ) =~ tr[\0][];
        if(  $sims == 19 ) {
            print "Match found: $kmers[ $i ], $kmers[ $j ]";
            push @varient_kmers, ($kmers[$i], $kmers[$j]);
        }
        ++$compare_count;
    }
}
print "\rFinished: $compare_count comparisions made.\n";

    [download]
    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

      shamshersingh:

      Here's what I tried:

      #!/usr/bin/perl
use strict;
use warnings;
my %H;
open my $FH, '<', 'DNA_strings.dat' or die $!;
while (<$FH>) {
    s/\s+$//;
    for my $i (0 .. length($_)-1) {
        my $k = $_;
        substr($k,$i,1) = '*';
        push @{$H{$k}}, $_;
    }
}

for my $k (sort keys %H) {
    if ($#{$H{$k}} > 1) {
        print "$k\t", join(",\n\t\t", @{$H{$k}}), "\n";
    }
}

      [download]

      A quick experiment with a short file:

      $ cat DNA_strings.dat
CTGAG
CGAGT
ACGCT
TATAC
CTGAA
GGAGC
ATACA
AAAAA
ACAAA
AGAAA
AATAA
AAAGA
ACCAA
AGCAC
CCACG
GCCAT
AGCAA
GGCAT
GTTTG

$ perl DNA_cmp.pl
A*AAA:    AAAAA, ACAAA, AGAAA
A*CAA:    ACCAA, AGCAA
AA*AA:    AAAAA, AATAA
AAA*A:    AAAAA, AAAGA
AC*AA:    ACAAA, ACCAA
AG*AA:    AGAAA, AGCAA
AGCA*:    AGCAC, AGCAA
CTGA*:    CTGAG, CTGAA
G*CAT:    GCCAT, GGCAT

      [download]

      It seems pretty fast, too. It took less than a minute to scan through 100,000 20-character strings, but it didn't find anything. (gen_random_strings.pl is on my my scratchpad)

      $ perl gen_random_strings.pl 100000 20 20 ACGT >DNA_strings.dat

$ time perl DNA_cmp.pl

real    0m47.659s
user    0m46.395s
sys     0m1.252s

      [download]

      Update: I tried to make this node a reply to the OP, but for some reason, it kept failing on me, and when I'd refresh, it would show my node replacing BrowserUk's node. It was odd looking...

      ...roboticus

      When your only tool is a hammer, all problems look like your thumb.

[reply]
[d/l]
[select]

        That's a brilliant way to do it. My own brute force method of comparing all the strings character by character required 5,000,050,000 comparisons and took almost 4 hours. Yours loops through 100,000 items once, then loops through a maximum of 2,000,000 keys once. It's not even close.

        Aaron B.
        My Woefully Neglected Blog, where I occasionally mention Perl.

[reply]

        I concur. This is extremely clever.

        It has the downside that it doesn't scale so well if you need to allow more than one char difference. But for the OPs stated problem, quite brilliant.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
Re: Comparing a large set of DNA sequences
by Anonymous Monk on Nov 10, 2011 at 05:15 UTC
    Thanks so much guys...you guys are the best
[reply]

Back to Seekers of Perl Wisdom