I've been looking around the Monestary this morning, but have not found what I need. memory usage woes and datastructures and out of memory errors come close.

I wrote a data conversion program that reads input from several files, munges it around, and spits it out in a single standard file format. (Standard for my company's internal systems, that is...)

In the munging process, I create a wide and shallow hash of hashes. I fill it something like this: 

$user_books{$user_id}{$ISBN} = $title;

[download]
I will have many ${user_id}s, each of which has an average of ten ${ISBN}s, each of which has exactly one ${title}.

I've been using it for several weeks, now, on relatively small input files - 'small' meaning 1,000 or fewer ${user_id}s. It's been very helpful where I work, and I'm getting questions along the lines of, "how many users can you convert, assuming the median number of books per user is ten?"

So, I need to know how much data I can handle. The only limitation I see is the amount of memory (real and virtual) available on the machine running the conversion, and this hash of hashes is the only long-lived thing of substantial size involved. Everything else is small, and scoped very narrowly.

I think I need to know how much memory each level of the hash requires, and then I can just do the math - how many records can I have before I run out of memory.
Am I asking the right question? If so, can anyone help me learn how to determine the memory requirements?

In reply to How to Calculate Memory Needs? by mrbbking

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.