Your code (and substr) works fine. I just generated a 2.6 MB test file, put START and END at the appropriate places and the results are exactly what you want.

That leaves a couple fo possibilities.

1. Either your method of checking is wrong.
2. The large file is not quite in the format your expecting.

BTW. You could replace your editcontig() sub with

  my $contigfile = "$contig.txt";
  open(CONTIG, $contigfile) 
      or die (">Could not open $contigfile!\n$!");
  my $goodseq = <CONTIG>;
  $goodseq =~ tr[0-9\n ][]d;


#clear giant variable to free some memory????
#  $rawseq = '';
[download]

This will reduce the memory required by your program from 35 MB to around 6MB.

It will also speed it up slightly, but if that offends you, you can always add a sleep 10; :)

  my $rawseq = <CONTIG>;
 
  # my $goodseq = editcontig($rawseq);
  $rawseq =~ tr/ \t\n0-9//d;

  # now use $rawseq instead of $goodseq
[download]

I don't think substr is your problem:

#!/usr/bin/perl

my $pad = 100_000_000;
my $s = 'a' x $pad;
$s .= 'qq was here';
$s .= 'b' x $pad;

print substr( $s, $pad, 10 ), "\n"; # qq was here
[download]

If you save the return from editcontig to a file, and find the character position that should be the start of the substr, is it actually off?

Its more likely that editconfig is stripping out more or less than it should. Perhaps there is some other character in your longer file? BTW, you can condense editconfig to something like:

sub editcontig {
  my $rawseq = shift;
  $rawseq =~ s/\s+//g;
  $rawseq =~ s/[0-9]//g;
  return $rawseq;
}
[download]

and you could validate the output by adding:

if ( $goodseq =~ /([^actg]+)/ ) {
  die 'found unexpected character $1";
}
[download]

qq (who knows nothing about genes)

s/[^atgc]+//g;
[download]

Since you're opening a file anyway, you might want to consider using the seek and read methods on your filehandle. This will allow you to position the read pointer anywhere you like within the file and read in exactly the number of bytes you require. Also, you will not need to read the entire file into memory before doing the string manipulation.

Update: The above suggestion would still work, but you would need to take into account the formatted nature of your input file. To use the seek method correctly, you would need to take into account the spaces between segments and the leading numbers prefixing each string (and likely the newline which terminates each string.) Using seek would still work, but depending on the format of the full input file, it might not be feasible.

Update II: (Thinking some more...) I have a question regarding the input file and the offset. Is the offset from the textual beginning of the file, or from the beginning of the DNA sequence? It would appear that it's from the beginning of the DNA sequence, but I'm not going to assume here.

The offset appears to make sense in the trimmed file. That is, excluding the spaces between the columns and the line numbers. Which suggests that Augustine might aviod slurping the file, by reading one line at a time (by leaving $/ unchanged) and windowing every input sequence (exon) (a sequence of lines which covers the exon).

You could also use `seek' and `sysread' (as suggested by Belgarion, but you'd have to work out an algorithm (quite simple I would guess) which converts the address within the trimmed file to the address within the formatted file. Something like... $address_in_file = $address_in_trimmed_file + $number_of_characters_for_line_numbering_for_first_line + ($address_in_trimmed_file / 10) + ($address_in_trimmed_file / 60) * ($one_character_for_new_line + $number_of_characters_for_line_numbering). Actually, I think that is a correct and yet unabusive way to solve this (every 10 characters you get a space, every 60 characters you get a new line and a line number).

On another note, though, I have experience with slurping large files, and can say that perl handles it pretty good. For example, I could semi-parse an 8Mb Xml file into an internal tree within seconds. I'm afraid I cannot say how substring handles large offset/length parameters, I thought it would work as expected.

Belgarion,

Thanks for your input. As I said I am pretty a green Perl man. I am not familiar with how to decide how many bytes converts into a given number of charaters so I would just assume stay away from that.

As for the input file, the textual begining of the file and the start of the DNA should be the same due to the editcontig sub or which ever of the new methods you guys suggested eating the numbers and the white space.

Thanks again, for your time I can tell this is where i need to ask questions from now on. The other boards i frequent are not 1/8 as helpful as you guys.

-Augustine

$rawseq =~ s/9//g;
[download]

Trek

As you are a green Perl hacker, I recommend two things:

1) Hang out here a lot. Share in the glow of our collective wisdom. Do a Super Search here in the Monastery for "DNA" (or even a short sequence like "agtc"!)and see what others who have gone before you have learned.

2) Get yourself a good Perl bioinformatics book. A quick search on Amazon yielded about a dozen or so. I have no idea which are worthwhile and which are junk, but I'm sure you are in a better position to judge. And, if in doubt, the O'Reilly books are usually a safe bet.

I am happy to say that the problem I had been having is now fixed. I am, however, embarrassed by what was causing the problem. I was careless when my boss asked me to find a region of the contig for her by hand and actually entered notes in the sequence file its self. Being that the file is 551 pages of pure text, I did not remember/notice the additions when I looked at the result of the editcontig() sub to look for problems.

Thanks again to all of you for your help and reminding me to verify anything that I edit before taking for granted that all went as planned.

I have been converted, evangelized even to a believer in the PerlMonk society and will most assuredly be bugging you nice people in the future!

one last thank you and I will end... THANK YOU!

-Augustine