This is yet another variation on the old chestnut of trying to trim down the memory usage of a long running Perl script. I have a long running (but not permanently running) process - a batch job, we'll call it - that analyzes financial data from the stock market. Specifically it looks at rows of stock data that consist of a date (a string), four floats, and an integer. It pulls this data from a database or a TokyoCabinet file, and a major source of overhead is this I/O, so it caches rows to reduce access to the disk. Because of the underlying processing being done, I can make very strong assumptions about how data in the cache will be used, and aggressively delete data that falls outside of a given time window. However, even with this deletion, the cache can end up consuming huge amounts of memory. Using Devel::Size and Devel::Size::Report, I have narrowed down the overhead on small arrays as a major culprit: 
Array ref 440 bytes (overhead: 232 bytes, 52.73%)
        Scalar 56 bytes
        Scalar 56 bytes
        Scalar 24 bytes
        Scalar 24 bytes
        Scalar 24 bytes
        Scalar 24 bytes

[download]
Each stored row has an overhead of 50% - half the memory being used by this application is array overhead! I have tried packing and unpacking the data to store it as scalars, but this causes performance to be even slower than without the cache. TIEing the arrays to disk would defeat the purpose of the cache. So - is there any way at all to reduce the overhead of these arrays that doesn't involve pack and unpack or deleting from the cache (which I'm already doing)?

In reply to bit by overhead by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.