in reply to Re^2: A question About Array Indexing
in thread A question About Array Indexing

A text file with a single line consisting of ~155e6 characters, all 0's and N's, is read in as a scalar variable. Each character is assigned to a single element of an array

As a scalar, your 155e6 file will require ~ 150 MegaBytes of memory.

As an array, 1 char per element, it will require ~ 10 GigaBytes of memory. Assuming you have that available.

But for your stated goal:

The goal: change all elements of the @info array that lie outside of my intervals to N's.

There is no need to go through the time and memory costly process of spliting your string to an array. And no need to iterate over 155 million characters one at a time.

You can easily and quickly overwrite the characters outside your ranges with 'N's, in-place in the scalar:

open( SEQ, 
    "/Users/logancurtis-whitchurch/Dropbox/thesis_folder/consensus_fil
+es/mask_files/mask."."$population".".chr.23.txt"
) or die "can't open masked file\n";

my $bigScalar = <SEQ>;
close SEQ;

open (INTERVAL, "<$filtered_sites") or die "can't open $filtered_sites
+";
my $lastEnd = 0;
while( <INTERVAL> ) {
    my( $start, $end ) = split "\t", $_;

    ## change everything from the end of the last range
    ## to the start of this range to 'N'
    substr( $bigScalar, $lastEnd, $start ) =~ tr[\x00-\xff][N]; 
    $lastEnd = $end;
}
close INTERVAL;

## change everything from the end of the last range to the end of stri
+ng to 'N'
substr( $bigScalar, $lastEnd, length( $bigScalar ) ) =~ tr[\x00-\xff][
+N];

## do something with $bigScalar
...

[download]
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^4: A question About Array Indexing
by ccelt09 (Sexton) on Aug 27, 2013 at 01:23 UTC

    This is great advice, and makes the process much more time and memory efficient! The one follow up question I have is pertaining to the line 

       
while( <INTERVAL> ) {
    my( $start, $end ) = split "\t", $_;

    [download]

    The text file being read has 3 columns, the first of which is non-numeric. If I specify the variables with 

    
foreach my $interval (<INTERVAL>){
            
    my @find_interval = split(/\t/, $interval);
    my $start = $find_interval[1]; 
    my $end = $find_interval[2];

    [download]

    would that accomplish the same thing?

[reply]
[d/l]
[select]
      foreach my $interval (<INTERVAL>){
       
     my @find_interval = split(/\t/, $interval);
     my $start = $find_interval[1]; 
     my $end = $find_interval[2];

      [download]

      Be aware that this loop will read the entire file accessed by the  INTERVAL filehandle into memory at once as an array, each line of the file being an array element. The
          while( <INTERVAL> ) { ... }
      loop reads and processes a line at a time: much more scalable, insignificant speed difference, if any.

      my @find_interval = split(/\t/, $interval);

      I would split directly into the named variables you will be using, and split on  '\s' (whitespace) to avoid having a newline stuck to the end of the third field element:
          my (undef, $start, $end) = split '\s', $_;

[reply]
[d/l]
[select]

      It would accomplish the same thing, yes. It may be more readable to use the former syntax if the array being split isn't huge however. So you are masking regions of the genome I take it?

      Bioinformatics
[reply]

        Yes indeed, just one chromosome for this program. The intervals are generally small but numerous in number since small interspersed sequences have been filtered out. 124467 intervals actually. It appears it will take a bit of time to go through all of them but I can't even imagine doing it the way I first proposed. Thank you for the help!

[reply]
      The text file being read has 3 columns, the first of which is non-numeric. If I specify the variables with foreach ... would that accomplish the same thing?

      Sorry, my mistake. However, I'd stick with while rather than foreach. There is simply no benefit to filling memory with an entire file (however big) if you can only use 1 line at a time: 

      while( <INTERVAL> ) {
     ## ignore the first field on each line
     my( undef, $start, $end ) = split "\t", $_;
...

      [download]
      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

In Section Seekers of Perl Wisdom