comment on

So after a few days of intermittent network connectivity(data is on a networked drive) and testing I figured it out. I think.

push(@array, $data);

I went down the path of using only 1 thread for processing. I went down the path of parsing only a single file type. Once I started the behavior went away, and of course it wasn't until the final file type that the behavior came back. I looked through my code to see what it had that no other file type had and it was...

@array = sort { $a <=> $b } @array;

So I took out that code and tested again, the problem was still present. So I kind of commented things out piece-meal until I narrowed it down to the single line of code above. With that single line of code 3 additional MB of memory is used up as the thread leaves the parsing method for that particular file type.

So here is the basic rundown of this file.

sub parsefiletypeX
{
      my $filename = shift;
      #get the directory from the filename(w/ directory)

      open(IN, $filename) or die...
      open(OUT, $outfile) or die...

      my $lineCount = 1;
      my $nextline = <IN>;
      my $headerlines;
      my $samplesize = 1;
      my @array;

      $nextline = trim_whitespace($nextline);#my subroutine
      ++$LineCount; #Did this before I learned about $.
      #read the header for the file(five lines)
      #Do a bunch of regex checks on the header lines

      #Read in the first record, which contains its own line of sub he
+ader data as well as the two lines of actual data in pascal float for
+mat I believe. Or fortran actually.

      #Regex on the first line and push a certain piece of data. This 
+line is NOT the bad line
      push( @array, @fi[4]);#1st line has 5 values
      #Regex checks on the next two lines

      #Then read in the rest of the 3-lined records
      until(eof(IN))
      {
            #get the same three lines and do the regex checks
            #now the faulty call is made
            push( @array, @fi[4]);
      }
      //Do stuff to the @array, like sorting and determining certain c
+hecks.
      close IN;
      close OUT;
      #call function that uploads a record to the database.
[download]

So after a few days of intermittent network connectivity(data is on a networked drive) and testing I figured it out. I think.

push(@array, $data);

@array = sort { $a <=> $b } @array;

So here is the basic rundown of this file.

sub parsefiletypeX
{
      my $filename = shift;
      #get the directory from the filename(w/ directory)

      open(IN, $filename) or die...
      open(OUT, $outfile) or die...

      my $lineCount = 1;
      my $nextline = <IN>;
      my $headerlines;
      my $samplesize = 1;
      my @array;

      $nextline = trim_whitespace($nextline);#my subroutine
      ++$LineCount; #Did this before I learned about $.
      #read the header for the file(five lines)
      #Do a bunch of regex checks on the header lines

      #Read in the first record, which contains its own line of sub he
+ader data as well as the two lines of actual data in pascal float for
+mat I believe. Or fortran actually.

      #Regex on the first line and push a certain piece of data. This 
+line is NOT the bad line
      push( @array, @fi[4]);#1st line has 5 values
      #Regex checks on the next two lines

      #Then read in the rest of the 3-lined records
      until(eof(IN))
      {
            #get the same three lines and do the regex checks
            #now the faulty call is made
            push( @array, @fi[4]);
      }
      //Do stuff to the @array, like sorting and determining certain c
+hecks.
      close IN;
      close OUT;
      #call function that uploads a record to the database.
[download]

I even tried clearing out that array after I was done using it, such as...

@array = ();
undef @array;
[download]

But this has no effect!?!? So what is going on?

In reply to Re^2: Multithreading leading to Out of Memory error by joemaniaci
in thread Multithreading leading to Out of Memory error by joemaniaci

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.