I'm parsing one input file for human chromosome locations, retrieve start and end position and then read in the chromosome file with scores to filter out a score at the specific locations - then calculate a mean score value for the entire region. The process is terribly inefficient, even if I split files. Each chromosome file is already around 3 GB. is there a better approach to this problem? Perhaps can I jump to the right locations in a speedier way? Help appreciated. 

open (F, $data) || die;
while(<F>) {

chomp;
if ($_!~/Header/g) {
@line=split("\t", $_);
$table=$_;
$chr=$line[0];
$start=$line[2];
$end=$line[3];

# file name
$name="score.txt";
@array=();
@sorted=();



#print "$chr\.$name\n"; 
open (R, "$chr\.$name") || die ("cannot open chrom file\n");

while(<R>) {
chomp;
# format : chrom ID, position, score
# chr1    10925   0.239

@q=split("\t", $_);
$pos=$q[1];
$score=$q[2];
#print "$pos\n"; 

if ($pos>=$start) {
    if ($pos<=$end) {
    
    
push (@array, $score);
$sum=$sum+$score;
}}

}
close(R);

# get mean min and max 
$len=$#array+1;

@sorted=sort {$a <=> $b} @array;
$min=$sorted[0];
$max=$sorted[$#sorted];
$mean=$sum/$len;

$stat = Statistics::Descriptive::Full->new();
$stat->add_data(@sorted);
$geo_mean= $stat->geometric_mean();

#print "$table\t$geo_mean\t$min\t$max\t$len\n"; 
print OUT "$table\t$mean\t$geo_mean\t$min\t$max\t$len\n";

}
}
close(F);
close(OUT);

[download]

In reply to speeding up parsing, jump to line by cburger

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.