http://qs321.pair.com?node_id=937241

shamshersingh has asked for the wisdom of the Perl Monks concerning the following question:

I have a large set (100000+) of short DNA reads 20 characters long. I need to compares all reads against each other and pull out those that vary by just 1 position. Heres the script I came up with.
$| = 1; my $compare_count = 0; for (my $i = 0; $i < @kmers; $i++ ) { for (my $j = $i + 1; $j < @kmers; $j++ ) { print "\rComparing sequence $i to $j"; my @result = PCCompare::dissimilarity($kmers[$i], $kmers[$j], +1); if ($result[0] == 1) { print "\rMatch found: $kmers[$i], $kmers[$j]\n"; push @variant_kmers, ($kmers[$i], $kmers[$j]); } $compare_count++; } } print "\rFinished: $compare_count comparisions made.\n";
The problem is that this loop runs very very slow. It takes on the orders of days to process 100000 sequences. Is there a way to make the process faster?

Replies are listed 'Best First'.
Re: Comparing a large set of DNA sequences
by BrowserUk (Patriarch) on Nov 09, 2011 at 23:09 UTC

    This will perform the same job, but run significantly faster (Approx 7 hours).

    $| = 1; my $compare_count = 0; for (my $i = 0; $i < @kmers; $i++ ) { for (my $j = $i + 1; $j < @kmers; $j++ ) { my $sims = ( $kmers[ $i ] ^ $kmers[ $j ] ) =~ tr[\0][]; if( $sims == 19 ) { print "Match found: $kmers[ $i ], $kmers[ $j ]"; push @varient_kmers, ($kmers[$i], $kmers[$j]); } ++$compare_count; } } print "\rFinished: $compare_count comparisions made.\n";

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      shamshersingh:

      Here's what I tried:

      #!/usr/bin/perl use strict; use warnings; my %H; open my $FH, '<', 'DNA_strings.dat' or die $!; while (<$FH>) { s/\s+$//; for my $i (0 .. length($_)-1) { my $k = $_; substr($k,$i,1) = '*'; push @{$H{$k}}, $_; } } for my $k (sort keys %H) { if ($#{$H{$k}} > 1) { print "$k\t", join(",\n\t\t", @{$H{$k}}), "\n"; } }

      A quick experiment with a short file:

      $ cat DNA_strings.dat CTGAG CGAGT ACGCT TATAC CTGAA GGAGC ATACA AAAAA ACAAA AGAAA AATAA AAAGA ACCAA AGCAC CCACG GCCAT AGCAA GGCAT GTTTG $ perl DNA_cmp.pl A*AAA: AAAAA, ACAAA, AGAAA A*CAA: ACCAA, AGCAA AA*AA: AAAAA, AATAA AAA*A: AAAAA, AAAGA AC*AA: ACAAA, ACCAA AG*AA: AGAAA, AGCAA AGCA*: AGCAC, AGCAA CTGA*: CTGAG, CTGAA G*CAT: GCCAT, GGCAT

      It seems pretty fast, too. It took less than a minute to scan through 100,000 20-character strings, but it didn't find anything. (gen_random_strings.pl is on my my scratchpad)

      $ perl gen_random_strings.pl 100000 20 20 ACGT >DNA_strings.dat $ time perl DNA_cmp.pl real 0m47.659s user 0m46.395s sys 0m1.252s

      Update: I tried to make this node a reply to the OP, but for some reason, it kept failing on me, and when I'd refresh, it would show my node replacing BrowserUk's node. It was odd looking...

      ...roboticus

      When your only tool is a hammer, all problems look like your thumb.

        That's a brilliant way to do it. My own brute force method of comparing all the strings character by character required 5,000,050,000 comparisons and took almost 4 hours. Yours loops through 100,000 items once, then loops through a maximum of 2,000,000 keys once. It's not even close.

        Aaron B.
        My Woefully Neglected Blog, where I occasionally mention Perl.

        I concur. This is extremely clever.

        It has the downside that it doesn't scale so well if you need to allow more than one char difference. But for the OPs stated problem, quite brilliant.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Comparing a large set of DNA sequences
by Anonymous Monk on Nov 10, 2011 at 05:15 UTC
    Thanks so much guys...you guys are the best