Two thoughts:

  1. If your program processes the files line-by-line, then the maximum memory it would need at any given time is the length of the longest line.

    Which for most files is a trivial amount.

  2. If you really need to load the files in their entirety each time, then slurping them into a single huge scalar rather than an array of lines, would ensure that when the file is processed and the scalar is freed, the whole amount of the scalar would be returned to the OS, not just the process pool.

    Note: I know this to be true of Perl running under Windows for single allocations over 1MB.

    The picture of whether other OS mallocs have similar arrangements for large, single allocations isn't so clear.

    Of course, this will only help if you can avoid breaking the single scalar into an array or hash.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

In reply to Re: Perl and memory usage. Can it be released? by BrowserUk
in thread Perl and memory usage. Can it be released? by sherab

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.