Yes, by all means, not only it can be done this way, but this is most often the best way you can do it (i.e. when the current data under examination doesn't need to know about the previous or next data chunk to be validated, examined or used). Especially with DNA files which, as far as I know(I am not a bio guy), can be very large. Using a while loop on your file (i.e. a file iterator) makes it possible for you to read absolutely huge files without ever running into out-of-memory problems (it might take time, but at least you have a very high probability of running your program to the end).

I am working almost daily with huge files (typically between 3 and 15 GB, sometimes as much as 200 GB). In such cases, slurping the file into memory is just not an option, my program would die. Reading it with an iterator (a 'while (my $line = <$IN>) {' type of construct) is the only solution, and it does not use more memory than the size needed for the longest line.

I had a relatively similar problem over the last days and found the solution this morning. A proprietary database (with no Perl module/driver), but implementing a protocol similar to SQL. I needed to load a rather large quantity of data into memory, and then process the main table using the data in memory. My first attempt last week ran out of memory and the program crashed. Before trying to load the main table, my original program was already using 155,000 blocks of memory (my best guess is that a memory block is 8 kB, but not sure). Anyway, after having loaded 155,000 blocks, trying to load the main table failed for lack of memory. After some experimentation, I was able to reduce memory consumption (changing hashes of hashes to hashes of strings), but the main improvement was to be able to use an iterator on the main table with a syntax as follows:

open my $invoice_lines, "-|", $command or die "could not fork $!";
while (my $inv_line = <$invoice_lines>) {
     #...
[download]

Other comments on your code: your identifiers are very poor, X, Y and A don't say anything about the content. Similarly, $a, $t, $g and $c may seem OK when talking about DNA, I would suggest you use at least two letters for better identification. Also, the $a (and also $b) variable has a special meaning in Perl (used especially for sorting) and should probably be avoided for other purposes.

Not sure I answered your question, but not sure about what your question really was. I hope that I gave at least some indications.

Dear All,

Thank you very much for your time and suggestions.

I agree Laurent R that these DNA files could be very long and loading them in an array at the beginning could pose a memory problems. This is the reason why I am trying to read the file in a while loop and checking for conditions. I tried to use if condition in the program

if (($seq =~/[A|T|G|C]/) && ($lenseq == 19))
    {
        print "$seq\n";
    }
    else {print "error log file";} # here I want to print those fragem
+nts whose length is either less than or greater than 19 and if the fr
+agments contains based other than [ATGC]
[download]

All this in a while loop so that I can read huge files without worrying about the memory issues.

Could it be possible to get some directions as to how to check those condition and only if the conditions are true the sequences are processed further.

Thanks to all of you

To check that a string contains something other than A, C, T, or G, search for the offending character, so in your condition, use

$seq !~ /[^ACTG]/
[download]

Note that | is not needed in a character class (in fact, it matches literally, so avoid it if you don't want to match it).

لսႽ† ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ

Hi, you could have a number of next statements to discard records that are not good. For example:

while (<$IN_FILE>) {
     chomp;
     next if /[^ACTG]/; # removes lines with other letters
     next if length != 19; # removes lines not 19 char long
     # I just made up the next rule for the example
     next if /(.)\1\1/; # removes lines where the same letter comes th
+ree times in a row
     # etc.
     # now start doing the real processing
     # ...
}
[download]

The next statement goes directly to the next iteration of the while loop, so that faulty lines are effectively discarded early in the process.

#!/usr/bin/env perl
use strict;
use warnings;

sub not_only_atgc {
    my $sequence = shift;
    if ( $sequence =~ /[^atgc]/i ) {
        return 1;
    }
    return 0;
}

my %test_criteria = (
    "Must not be 19 characters long" => sub {
       return 1 if (length($_[0]) != 19);
    },
    "Must contain other than atgc" => \&not_only_atgc,
);

my $lineno = 0;

# Read one line at a time
open( my $fh, "data.txt" ) or die "$!";
LINE:
while ( my $line = <$fh> ) {
    $lineno++;
    my @words = split /\s+/, $line;
    print "Processing line $lineno interested in $words[2]\n";

    TEST:
    for my $criteria_check ( keys %test_criteria ) {
        if ( $test_criteria{$criteria_check}->( $words[2] ) ) { # Test
+ returned 1
        } else {
           print "Fail: $criteria_check\n\n";
           next LINE;
        }
    }

    ## All checks have passed after this line ##
    print "$words[2] passes all criteria\n\n";
}
[download]

Processing line 1 interested in caggctcaggacttagcaa
Fail: Must contain other than atgc

Processing line 2 interested in cttagcaagaagttatgaaa
Fail: Must contain other than atgc

Processing line 3 interested in ggcycaggacttagcaaga
Fail: Must not be 19 characters long

Processing line 4 interested in caggacttagcaaoooaagtt
caggacttagcaaoooaagtt passes all criteria
[download]