So after a few days of intermittent network connectivity(data is on a networked drive) and testing I figured it out. I think.

I went down the path of using only 1 thread for processing. I went down the path of parsing only a single file type. Once I started the behavior went away, and of course it wasn't until the final file type that the behavior came back. I looked through my code to see what it had that no other file type had and it was...

So I took out that code and tested again, the problem was still present. So I kind of commented things out piece-meal until I narrowed it down to the single line of code above. With that single line of code 3 additional MB of memory is used up as the thread leaves the parsing method for that particular file type.

So here is the basic rundown of this file.

sub parsefiletypeX
{
      my $filename = shift;
      #get the directory from the filename(w/ directory)

      open(IN, $filename) or die...
      open(OUT, $outfile) or die...

      my $lineCount = 1;
      my $nextline = <IN>;
      my $headerlines;
      my $samplesize = 1;
      my @array;

      $nextline = trim_whitespace($nextline);#my subroutine
      ++$LineCount; #Did this before I learned about $.
      #read the header for the file(five lines)
      #Do a bunch of regex checks on the header lines

      #Read in the first record, which contains its own line of sub he
+ader data as well as the two lines of actual data in pascal float for
+mat I believe. Or fortran actually.

      #Regex on the first line and push a certain piece of data. This 
+line is NOT the bad line
      push( @array, @fi[4]);#1st line has 5 values
      #Regex checks on the next two lines

      #Then read in the rest of the 3-lined records
      until(eof(IN))
      {
            #get the same three lines and do the regex checks
            #now the faulty call is made
            push( @array, @fi[4]);
      }
      //Do stuff to the @array, like sorting and determining certain c
+hecks.
      close IN;
      close OUT;
      #call function that uploads a record to the database.
[download]

,

So after a few days of intermittent network connectivity(data is on a networked drive) and testing I figured it out. I think.

I went down the path of using only 1 thread for processing. I went down the path of parsing only a single file type. Once I started the behavior went away, and of course it wasn't until the final file type that the behavior came back. I looked through my code to see what it had that no other file type had and it was...

So I took out that code and tested again, the problem was still present. So I kind of commented things out piece-meal until I narrowed it down to the single line of code above. With that single line of code 3 additional MB of memory is used up as the thread leaves the parsing method for that particular file type.

So here is the basic rundown of this file.

sub parsefiletypeX
{
      my $filename = shift;
      #get the directory from the filename(w/ directory)

      open(IN, $filename) or die...
      open(OUT, $outfile) or die...

      my $lineCount = 1;
      my $nextline = <IN>;
      my $headerlines;
      my $samplesize = 1;
      my @array;

      $nextline = trim_whitespace($nextline);#my subroutine
      ++$LineCount; #Did this before I learned about $.
      #read the header for the file(five lines)
      #Do a bunch of regex checks on the header lines

      #Read in the first record, which contains its own line of sub he
+ader data as well as the two lines of actual data in pascal float for
+mat I believe. Or fortran actually.

      #Regex on the first line and push a certain piece of data. This 
+line is NOT the bad line
      push( @array, @fi[4]);#1st line has 5 values
      #Regex checks on the next two lines

      #Then read in the rest of the 3-lined records
      until(eof(IN))
      {
            #get the same three lines and do the regex checks
            #now the faulty call is made
            push( @array, @fi[4]);
      }
      //Do stuff to the @array, like sorting and determining certain c
+hecks.
      close IN;
      close OUT;
      #call function that uploads a record to the database.
[download]

I even tried clearing out that array after I was done using it, such as...

@array = ();
undef @array;
[download]

But this has no effect!?!? So what is going on?