Hi,

I am a geneticist/biochemist and trying to use Perl to analyze DNA sequence data. I have a giant text file (99 million lines; ~ 7GB). The file contains DNA sequences. I need to extract duplicate lines based on the BeadIDs from input file and print it into different file. BeadIDs are written in field one (first column), for example followings are BeadIDs:

99_99_992
99_999_930
99_999_930
99_99_999   
99_99_999
99_99_9992
99_99_9992
99_99_9992
[download]

Real Input file looks like this (total ~99 million lines):

99_99_992    chr16.fsa    11064    11088    TAGGTAAACGAGGAGTCTTTCAATA 
+   -    25
99_999_930    chr04.fsa    148776    148800    TGCAAACGAGGACAACCTGTTTG
+TG    +    25
99_999_930    chr04.fsa    148882    148916    CCGGAAAAATTTGCTATTGGAAG
+AGGTGGCGCTGG    -    35
99_99_999    chr12.fsa    468017    468049    TTTTCGGTGACGGAAATACGCTTC
+AGAGACCCT    +    33
99_99_999    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTCT
+A    -    25
99_99_9992    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTC
+TA    -    25
99_99_9992    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTC
+TA    -    25
99_99_9992    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTC
+TA    -    25
...
[download]

The output that I would like to have is shown below (BeadIDs must be repeated exactly TWICE, not unique, not triplicate, not quadruplicate, but I want the entire line not only column one):

99_999_930    chr04.fsa    148776    148800    TGCAAACGAGGACAACCTGTTTG
+TG    +    25
99_999_930    chr04.fsa    148882    148916    CCGGAAAAATTTGCTATTGGAAG
+AGGTGGCGCTGG    -    35
99_99_999    chr12.fsa    468017    468049    TTTTCGGTGACGGAAATACGCTTC
+AGAGACCCT    +    33
99_99_999    chr12.fsa    468138    468162    CTATGTTTTCTTACTCCTATGTCT
+A    -    25
[download]

I have written a code that would choose duplicate lines if my input file had only single column(field). However, as you can see my actual file has 7 fields, and I need to choose the duplicate lines based on first field, and print the entire line. Sor far, I could come up with following script that would work on a single column file. However, my Perl knowledge is not advance enough to come up for a solution when input file is in 7 field format.

My code so far:

 
#!/usr/bin/perl
#use strict;
use warnings;

$numArgs = $#ARGV + 1;

if ($numArgs < 2)
{
    print "USAGE: extract_duplicate_rows <input file> <# of duplicate 
+rows>\n\n";
    exit(0);
}

$finput = $ARGV[0];
$duplicate_count = $ARGV[1];


open(IFILE, "<", $finput) or die $!;

chomp($prev_line = <IFILE>); 
$match_count = 1;

while (1)
{
    chomp($line = <IFILE>); 
    
    if ($line eq $prev_line)
    {
        $match_count++;
    }
    else
    {
        if ($match_count == $duplicate_count)
        {    
            print "$prev_line\n";
        }

        $prev_line = $line;
        $match_count = 1;

        if (eof(IFILE))
        {
            last;
        }        
    }
}

    
close IFILE;
[download]

I was wondering if you guys could make me suggestions. Thanks for your help. ~Erbay

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.