You haven't said what kind of processing you're doing, but, in the vast majority of cases, you don't actually need to load the whole file into memory. If the files have any kind of internal structure, you can probably read them in corresponding chunks - whether that's line-by-line or record-by-record - to do your processing with only one chunk (or maybe a small number of chunks, depending on the nature of your processing) in memory at a time.

If your processing amounts to "read in the whole file, then loop over pieces of the file to handle each piece in turn", then you almost certainly do not need to read it all in at once.

In reply to Re: Why I got nearly out of memory, and never recover from that? by dsheroh
in thread Why I got nearly out of memory, and never recover from that? by llancet

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.