I think arhuman definitly has the right idea. If you are looking for speed and want to save memory by not loading the entire file into memory, saving an index and using seek() is definitly going to be a good bet. What you may want to do is save the database file sorted by the primary keys, and use the Binary Search Algorithm's from Mastering Algorithms with Perl, this could reduce the amount of data in the index by allowing faster searches through the data.

Another thing to keep in mind is some sort of query optimization. By this i mean requests need to be screened to see if the will require a complete table scan, and in the case it may be worth reading in the entire file (rather than line-by-line with a seek, you could make it one huge sequential IO operation). Well, not too much new in this, but just some food for thought.

may the foo be with you

In reply to Re: The Never-ending Database by MZSanford
in thread The Never-ending Database by psypete

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.