extracting duplicates from a list

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks, I am trying to extract unique sequences (and there identifiers) from a list like this:


>atc:AGR_pTi_39_1-45_FD cctttcaagtcatagaacaccggggcatgtacaacttggggaagg
>atc:AGR_pTi_47_1-45_FD ccttacaggtcattgagcacagaggaatgttcaatttagggaaac
>atc:AGR_pTi_39_1-45_F      cctttcaagtcatagaacaccggggcatgtacaacttgggga
+agg
>atc:AGR_pTi_47_1-45_F      ccttacaggtcattgagcacagaggaatgttcaatttaggga
+aac
>atc:AGR_pTi_39_1-45_RD cctttcaagtcatagaacaccggggcatgtacaacttggggaagg
>atc:AGR_pTi_47_1-45_RD ccttacaggtcattgagcacagaggaatgttcaatttagggaaac
>atc:AGR_pTi_39_1-45_R  cctttcaagtcatagaacaccggggcatgtacaacttggggaagg
[download]

I am just comparing the sequences part of each but wish to extract the sequence and the annotation of those that are unique. I have written a script which extracts unique sequences but am failing to extract the annotation. Please can anyone help.

# my code so far

#!/usr/bin/perl -w
                                                                      
+                                               
open (FILEHANDLE, $ARGV[0]) or die "unable to open file";
open (OUTFILE, ">$ARGV[1]") or die;                                   
+                                                                     
+             
my $line;
my @array;                                                            
+                                                         
my @seqs;
                                                                      
+                                               
while (<FILEHANDLE>)
{
    $line = $_;
    chomp ($line);                                                    
+                                                                 
                                                                      
+                                               
    if ($line =~ /(\S+)(\s+)(\w+)/)   {                               
+                                                                     
+                 
            push @seqs, "$3 ";                                        
+                                                                     
+        
    }                                                                 
+                                                                     
+                                                                     
+                               
}
                                                                      
+                                               
foreach my $item (@seqs)   {
    unless ($seen{$item})   {
          $seen{$item} = 1;
          push (@uniq, $item);
          push @uniq, "\n";
 }                                                                    
+                                                                     
+                                                                     
+                            
    else {                                                            
+                                                         
          push (@duplicates, $item);
          push @duplicates, "\n";                                     
+                                                                     
+           
    }
}
                                                                      
+                                               
print @uniq;                                                          
+                                                           
print OUTFILE "@duplicates\n";                                        
+                                                                     
+        
close FILEHANDLE;
close OUTFILE;
exit;
[download]

BazB added readmore tags

Comment on extracting duplicates from a list Select or Download Code

Replies are listed 'Best First'.
Re: extracting duplicates from a list by kvale (Monsignor) on Feb 20, 2004 at 16:55 UTC
Here is code that parses your data format and flags sequences that are unique in your data. The `%count` hash can also be used to process duplicates. #!/usr/bin/perl -w use strict; my (%count, %ann); while (my $line = <DATA>) { chomp $line; if ($line =~ /^>atc:(\S+)\s+(\w+)$/) { $count{$2}++; $ann{$2} = $1; } } foreach my $seq (keys %count) { print "Unique seq $seq has ann $ann{$seq}\n" if $count{$seq} == 1; } __DATA__ >atc:AGR_pTi_39_1-45_FD cctttcaagtcatagaacaccggggcatgtacaacttggggaagg >atc:AGR_pTi_47_1-45_FD ccttacaggtcattgagcacagaggaatgttcaatttagggaaac >atc:AGR_pTi_39_1-45_F cctttcaagtcatagaacaccggggcatgtacaacttgggga +agg >atc:AGR_pTi_47_1-45_F ccttacaggtcattgagcacagaggaatgttcaatttaggga +aac >atc:AGR_pTi_39_1-45_RD cctttcaagtcatagaacaccggggcatgtacaacttggggaagg >atc:AGR_pTi_47_1-45_RD ccttacaggtcattgagcacagaggaatgttcaatttagggaaac >atc:AGR_pTi_39_1-45_R cctttcaagtcatagaacaccggggcatgtacaacttggggaagg [download] -Mark	[reply] [d/l] [select]
Re: extracting duplicates from a list by falsa_utopia (Initiate) on Feb 20, 2004 at 21:26 UTC
In case you care about the different tags on each sequence: #!/usr/bin/perl -w use strict; my %unique = ( ); while (<DATA>) { chomp; my ($tag, $sequence) = split; next unless defined $tag; if (exists $unique{$sequence}) { push @{ $unique{$sequence} }, $tag; } else { $unique{$sequence} = [ $tag ]; } } foreach (keys %unique) { print "Sequence: $_\n\tTags:\n"; foreach my $tag (@{ $unique{$_} }) { print "\t$tag\n"; } } __DATA__ >atc:AGR_pTi_39_1-45_FD cctttcaagtcatagaacaccggggcatgtacaacttggggaagg >atc:AGR_pTi_47_1-45_FD ccttacaggtcattgagcacagaggaatgttcaatttagggaaac >atc:AGR_pTi_39_1-45_F cctttcaagtcatagaacaccggggcatgtacaacttgggga +agg >atc:AGR_pTi_47_1-45_F ccttacaggtcattgagcacagaggaatgttcaatttaggga +aac >atc:AGR_pTi_39_1-45_RD cctttcaagtcatagaacaccggggcatgtacaacttggggaagg >atc:AGR_pTi_47_1-45_RD ccttacaggtcattgagcacagaggaatgttcaatttagggaaac >atc:AGR_pTi_39_1-45_R cctttcaagtcatagaacaccggggcatgtacaacttggggaagg [download] HTH	[reply] [d/l]
Re: extracting duplicates from a list by pelagic (Priest) on Feb 20, 2004 at 16:42 UTC
Why don't you just split the lines. They look like having two words with a number of blanks inbetween each. pelagic ------------------------------------- I can resist anything but temptation.	[reply]