using a perl LoL and almost running out of memor

ai56 has asked for the wisdom of the Perl Monks concerning the following question:

Hi again, I have program that is initializing a perl LoL (array of arrays) by reading in comma separated lines from 20 different files and then storing contents in a 2-d array. I am clearing out the data structure at the end of each iteration so I don't adding rows to the array. But the program keeps allocating memory (almost but not quite running out). The program finished, but then the perl process is killed, the memory is not freed. I don't know if its something I'm doing with manipulating the 2-d matrix that is causing the memory leak. I did look at the the perlman:perllol doc which helped. But I think I might be doing something wrong still. Any help is appreciated. The code snippet is here:

# read in lines across 20 files
while (1) {
my $batch1 = <FILE1>; chomp($batch1); 
last unless ($batch1);
...
my $batch20 = <FILE20>; chomp($batch4);

# put the lines into an array
# @batch_lines has 20 elements
push(@batch_lines, $batch1);
...
push(@batch_lines, $batch20);


#init the 2-array @confidence_mtx after parsing out some #data for the
# batch_lines elements

foreach (@batch_lines) {
  next if  /^Probeset/;
  my ($probe, $tmp) = split /\t/;
  push (@confidence_mtx, [ split /,/ , $tmp ]);
}

# do some lookups in the confidence_mtx data structure
my $batchnumber = $sample2batch{$consensus_sample_batch};
my $offset = $sample2offset{$consensus_samples_offset};

if ( $confidence_mtx[$batchnumber-1][$offset-1] > $CONFIDENCE_CUTOFF) 
+{
  $consensus_genotypes[$i] = '00';
}

undef @confidence_mtx;

} # close while(1) loop
[download]

Comment on using a perl LoL and almost running out of memor Download Code

Replies are listed 'Best First'.
Re: using a perl LoL and almost running out of memor by ikegami (Patriarch) on Oct 23, 2007 at 01:54 UTC
Where's `@batch_lines` cleared? Add `my @batch_lines;` in the `while` loop.	[reply] [d/l] [select]
Re: using a perl LoL and almost running out of memor by kyle (Abbot) on Oct 23, 2007 at 02:35 UTC
I find your question confusing, so I don't have an answer, but it reminds me of Strings and numbers: losing memory and mind., which I wrote a few weeks ago. In that case, my program continued to allocate memory after a large array was populated, even though I wasn't adding any more elements to the array. This turned out to be because the elements of the array were being converted from strings to numbers on the fly, and Perl would keep both representations for each one. The solution for me was to explicitly make them numbers as I was reading them in (so instead of putting `$x` into the array, put in `0+$x` so it's a number). As I say, this might not be your problem, but if it is, my hope is that you wouldn't have to spend as much time figuring it out as I did.	[reply] [d/l] [select]
Re: using a perl LoL and almost running out of memor by Thilosophy (Curate) on Oct 23, 2007 at 01:55 UTC
`undef @confidence_mtx;` [download] How about `@batch_lines`? Are you also clearing that one? The program finished, but then the perl process is killed, the memory is not freed. That sounds unlikely. After the perl process is finished, all its resources (including memory) should be freed by the OS.	[reply] [d/l] [select]