Here is basically where I have pinpointed the memory leak to (caution: I don't know if this code will actually work, its just a snippet from my program): 
$linectr=0;
$ctr2=0;

for $line (@array){
  $hash{$linectr}=$line;
  if($ctr2=100000){
    addtofile(\%hash);
    %hash=();
    $ctr2 = 0;
  }
  $ctr2++; $linectr++;
}
sub addtofile {
  my $hashref = shift;
  open(FH,$tempfile);
  foreach $value (keys %$hashref) {
    print FH "$$hashref{$value}:$value\n";
  }
  close FH;
}

[download]
when I close the FH, some memory gets deallocated, and the RSS size on top goes down, but not completely, so it keeps growing and growing. on Linux this doesn't happen very fast, so my program usually finishes before I run out of memroy, but on OSX the memory grows by leaps and bounds, and I run out of memory about 10-15 passes through the program.

however, I found a solution to my problem using Berkeley DB. Since I am trying to sort files that are too large to put in memory, I thought I would have to break them up and use a merge-sort algorithm, but if I tie an empty file to the Berkeley DB object I can treat the file as an array and insert lines into the middle of the file, so no need to keep opening and closing files.

janitored by ybiC: balanced <code> tags as per Monastery convetion, and a bit o'formatting

In reply to Re: Re: cleaning up memory after closing files on MACOSX by codingchemist
in thread cleaning up memory after closing files on MACOSX by codingchemist

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.