I am encountering a very irritating situation with a group of DNA strings saved in a special file format with the FastA extension. While this format is extremely simple and its files can readily be opened by ol' NotePad I still find that my program sort of takes a very long time to complete.

What I am trying to do is open the FastA file that has these DNA sequences, read the descriptor from the headers for each sequence and then use the descriptor as a new filename and output the sequences to these files accordingly in preparation for further analyses, (N.B that can be taken care of in due time). 

#An example of my data file 'Test.Fasta', which has 3 records starting
+ at '>'
>gi|62750809
TGAGCATGGGAGATCTTTCCATCTTCTGAGGTCTTCTTCAATTTCTTTCCTCAGTGTCTTGAAGTTCTTA
TTGTACAGATCTTTTACTTGCTTGGTTAATGTCACACCGAGGTATTTTATATTATTTGGGTCTATTATGA
>gi|151301097
TTTCTGGCTCCGCGGGCAGCGGGGCCGTGGCGCTCGGACGGTCTGGGATTCGGGCGCCGCCGCGGAACCG
GAATAAGAAGGGAGAGCGCCCGGCTCGGTCCTCGGTCTCCACCGCGGCCCGGAAGGAATCCGGGCAGCCT
>gi|25266387
TGTGTATGTATATTAATTACATTCATATGTATTCACAACACCTGCCTCAAATCAGGCAGAATGGTCCAGG
ATGGAATTAGGGGCAAGCATGAGGTCTTCAGGCTTACTGATTTCTAAGACACAGTAACTTCACTGGTTAG

[download]
of Course each one of these records is very big that it spans a lot of lines and there are many records that the entire file is >200,000 KBs. I wrote the following program in BioPerl and just to get the program to print the descriptors after '>' takes more than a half hour whereas even if I turned flushing on while writing to a file the program takes forever again so monks tell me what can be a potential cause of this, and what measures do you implement when dealing with huge files. I am running on an HP with 2 GB of RAM and a duo core processor ...

Here's my code (scaled down to represent the problem) 

#!/usr/local/bin/perl
use strict;
use warnings;
use Bio::SeqIO;

my $file = "Test.fasta";
my $in = Bio::SeqIO->new(
        -file => "<$file",
        -format => 'fasta',
        -flush=>0
        );

while (my $seq=$in->next_seq){
        my $desc = $seq->desc;
        print $desc, "\n" ;
        }

[download]

In reply to Bioinformatics: Slow Parsing of a Fasta File by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.