Dear Monks,
I have been maintaining quite a big object oriented application for three years now. This application has probably a person year of development under its hood. While everything was fine for a long time, suddenly the requirements have changed. The data to be stored has increasd by large and while I had several cases where the memory consumption was closing in on the half a gigabyte mark, I have some test cases where it exceeds 2 Gbyte and subsequently crashes. The data used in this applciation is in a text file and merely 200 MByte large. While I am interested in maintaining the structure of this application.
I would like to revisit my classes and make the data storage more memory efficient. Ideally this does not mean to revert to I/O but instead to store/access data differently. I seem to have heard that a scalar requires at least 32 byte in Perl. As a large number of my values is boolean or very small intergers I am considering using a central scalar with an integer value that can be accessed as a bitfield. Similarly several short strings could be joint to one string and accessed using substring or unpack.
Before I can analyze which way to follow and how small or big the possible benefit is, I need to understand how much memory is used for what in perl. I. e. there is no value optimizing the data storage if the real bottleneck is caused by the amount of accessors to the objects values. I have googled for sources and hints how to reduce memory consumption for several days now but so far I have not found any sensible solutions. Any help is greatly appreciated.

Cheers,
PerlingTheUK

In reply to Optimizing Memory consumption by PerlingTheUK

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.