in reply to Re: A question About Array Indexing
in thread A question About Array Indexing

I apologize for the poorly constructed explanation and I do appreciate your help very much. A text file with a single line consisting of ~155e6 characters, all 0's and N's, is read in as a scalar variable. Each character is assigned to a single element of an array:

open(INPUT, "/Users/logancurtis-whitchurch/Dropbox/thesis_folder/conse +nsus_files/mask_files/mask."."$population".".chr.23.txt") or die "can +'t open masked file\n"; ;; my $input = <INPUT>; ;; chomp($input); ;; my @info = split( //, $input );

I'd like to read each element of @info sequentially, at the same time have a scalar $length that corresponds to the length from the beginning of @info, $info[0], to the current, farthest position being read, say $info[3000] where $length would equal 3001. With this information I can tell the program, if $length is within the bounds of my current interval values ( $start and $end) go on to the next element of @info and increment $length and so on until $length is greater than $end, in which case I increment my interval values which have been similarly assigned to an array from an input text file. Then I would like to tell the program if $length is less than $start change the current element of @info to an 'N' , move to the next element of @info and increment $length. The goal: change all elements of the @info array that lie outside of my intervals to N's.

this shows the format of my interval input file and how I assign $start and $end values:

input format for intervals: chrX 1 6000 chrX 6045 6302 chrX 7204 8239 ...etc until all 155e6 positions have been covered. my $filtered_sites = "/Users/logancurtis-whitchurch/Dropbox/thesis_fol +der/galaxy_chrX_data/filtered_chrX_rawdata.interval"; ;; open (INTERVAL, "<$filtered_sites") or die "can't open $filtered_sites +"; ;; my @interval = <INTERVAL>; ;; close (INTERVAL); ;; chomp (@interval); ;; my @site_info= split(/\t/, $interval[$count]); ;; my $start = $site_info[1]; ;; my $end = $site_info[2]; ;;

so every time $length is greater than $end I increment $count to assign new $start and $end values.

Replies are listed 'Best First'.
Re^3: A question About Array Indexing
by BrowserUk (Patriarch) on Aug 26, 2013 at 23:57 UTC
    A text file with a single line consisting of ~155e6 characters, all 0's and N's, is read in as a scalar variable. Each character is assigned to a single element of an array

    As a scalar, your 155e6 file will require ~ 150 MegaBytes of memory.

    As an array, 1 char per element, it will require ~ 10 GigaBytes of memory. Assuming you have that available.

    But for your stated goal:

    The goal: change all elements of the @info array that lie outside of my intervals to N's.

    There is no need to go through the time and memory costly process of spliting your string to an array. And no need to iterate over 155 million characters one at a time.

    You can easily and quickly overwrite the characters outside your ranges with 'N's, in-place in the scalar:

    open( SEQ, "/Users/logancurtis-whitchurch/Dropbox/thesis_folder/consensus_fil +es/mask_files/mask."."$population".".chr.23.txt" ) or die "can't open masked file\n"; my $bigScalar = <SEQ>; close SEQ; open (INTERVAL, "<$filtered_sites") or die "can't open $filtered_sites +"; my $lastEnd = 0; while( <INTERVAL> ) { my( $start, $end ) = split "\t", $_; ## change everything from the end of the last range ## to the start of this range to 'N' substr( $bigScalar, $lastEnd, $start ) =~ tr[\x00-\xff][N]; $lastEnd = $end; } close INTERVAL; ## change everything from the end of the last range to the end of stri +ng to 'N' substr( $bigScalar, $lastEnd, length( $bigScalar ) ) =~ tr[\x00-\xff][ +N]; ## do something with $bigScalar ...

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

      This is great advice, and makes the process much more time and memory efficient! The one follow up question I have is pertaining to the line

      while( <INTERVAL> ) { my( $start, $end ) = split "\t", $_;

      The text file being read has 3 columns, the first of which is non-numeric. If I specify the variables with

      foreach my $interval (<INTERVAL>){ my @find_interval = split(/\t/, $interval); my $start = $find_interval[1]; my $end = $find_interval[2];

      would that accomplish the same thing?

        foreach my $interval (<INTERVAL>){ my @find_interval = split(/\t/, $interval); my $start = $find_interval[1]; my $end = $find_interval[2];

        Be aware that this loop will read the entire file accessed by the  INTERVAL filehandle into memory at once as an array, each line of the file being an array element. The
            while( <INTERVAL> ) { ... }
        loop reads and processes a line at a time: much more scalable, insignificant speed difference, if any.

        my @find_interval = split(/\t/, $interval);

        I would split directly into the named variables you will be using, and split on  '\s' (whitespace) to avoid having a newline stuck to the end of the third field element:
            my (undef, $start, $end) = split '\s', $_;

        It would accomplish the same thing, yes. It may be more readable to use the former syntax if the array being split isn't huge however. So you are masking regions of the genome I take it?

        Bioinformatics
        The text file being read has 3 columns, the first of which is non-numeric. If I specify the variables with foreach ... would that accomplish the same thing?

        Sorry, my mistake. However, I'd stick with while rather than foreach. There is simply no benefit to filling memory with an entire file (however big) if you can only use 1 line at a time:

        while( <INTERVAL> ) { ## ignore the first field on each line my( undef, $start, $end ) = split "\t", $_; ...

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.