If you can make your data look like a hash, you can use something like BerkeleyDB, which is quite good at handling large amounts of data (with caching, etc.). I wouldn't use an RDBMS unless I had relations (i.e. more than one table) or a need for query language. You don't have either one of these.

By tuning the page size and cache size, and giving other hints to BerkeleyDB, you can get very good performance, and you can whip up something to try very quickly. I'd suggest using BerkeleyDB through its hash interface, and using a BTree, and then seeing if it's fast enough. The results may surprise you.

In reply to Re: How to best handle memory intensive operations by bikeNomad
in thread How to best handle memory intensive operations by JPaul

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.