Hello,

I have a programm, that requires text files to be read into memory. The size of these files started as "small" 5 megabyte ones but has increased to about 125 megabytes. Loading these files into objects of my class structure requires about 1.5 GB of memory.
I cannot read only parts of this file as the data needs to be sorted in a very complicated way.
I now try to reduce the memory required. Unfortunately there seems to be little information about how to do so. I tried a couple of things so far:
  1. The text file I read, has fixed columns sizes, instead of splitting these into my object hash's key-value pairs, I just stored the lines into my objects using the substr function for all get-methods. I did that because I read that a scalar, requires a minimum amount of memory that would exceed the length of most of my values size. The effect was an obviously increased runtime as substr now was called about 16 million times per object, but no remarkable reduction in memory usage, which made me conclude that the minimum size required for a hash's key-value pair is much closer to zero than of ?32? Bytes for a scalar.
  2. I also tried limiting memory usage by removing useless whitespaces within a value, but there are few and memory reduction was less than 2 percent.
  3. My last and more successful effort was to reduce memory by leaving all values that have a standard value as undefined and just return the standard value. This reduced memory by about ten percent but still that is much less than required.
Are there any rescources about reducing memory usage out there? Anything that makes me understand how memory is allocated in scalars/arrays/hashes? Or can I possibly reduce the usage by enconding my values into strings using a wider range of characters and reducing the length of a string but still enabling me to sort those elements without decoding the values as this would very likely take too much time again.

Thanks.

In reply to Reducing Memory Usage by PerlingTheUK

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.