short scenario: I have noticed with perl on OSX that when I close a filehandle, the memory does not get completely deallocated, so when my perl script opens and closes lots of files (extremely large files) the memory doesn't get deallocated, the VSIZE keeps growing, and eventually I get a malloc error. Any ideas?

long detailed scenario: I am sorting extremely large files (>2Gig) on a MACOSX. I take a chunk of the file (usually about 100000 lines) sort them, store them in a tempfile. take another chunk of the file, sort those lines, and then merge them with the tempfile to another tempfile. repeat ad nauseum. I use two temp files and toggle between reading and writing to them. what I am doing now, is everytime I want to merge my sorted lines with a tempfile I open both temp files (one for reading, one for writing) and then close both of them before grabbing the next chunk of lines. for some reason closing the files isn't cleaning out memory the way it should, so if I keep this up, I get a malloc error, and then the system panics (note: this doesn't happen on linux). I am trying to work it so I can leave the files open, but on OSX the O_RDWR|O_CREAT flags for sysopen aren't actually letting me read the file, just write to it, same with the +>> for open. any ideas?

In reply to cleaning up memory after closing files on MACOSX by codingchemist

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.