I had a similar problem 2 years ago. My solution was to cook up a weird disk-based hashing method. Each of my records had a primary identifier (a unique name) which I converted to ascii and summed the integers. I then multiplied by a really large prime number (for fun and further diversity) and then I added the ascii values for the first and last characters again and then did modulo division with 65536. The result for each record was a pretty quickly computed bucket for each record which was nearly random. I converted the remainder to hex so I'd get filenames between 00/00.txt .. FF/FF.txt ... up to 256 directories each with up to 256 text files. I added many hundreds of thousands of records and it was still wonderfully fast but it would have been horrendous if they'd all resided in a single file. If you don't have a unique key for each record, this solution would not be helpful but it werked really well for me back in the day.

Since then, I've picked up the O'Reilly MySQL & mSQL book and have lerned DBI. If you have permissions to setup a daemon on a server, it's really pretty painless. I've compiled MySQL a couple of times now and installed 3 or 4 different versions (at different times) without too much headache. SQL is like Perl in that it is a rich language but it's okay not to know everything immediately. There are a few key things that can be lerned quickly and you can get by with those... the rest just makes you better. SQL is easy enough and DBI seems to provide the portability and flexibility to plug in differing database engines and drivers rather simply. I've been won over to relational databases since they are really powerful and they're not as hard as I thought they'd be to use (maybe MySQL is a cakewalk compared to Oracle but it's a start). I hope this helps somewhat. I'd be glad to share more of my flatfile madness if you'd like. Shalom.

-PipTigger

p.s. Initiate Nail Removal Immediately!

In reply to Re: DBI vs MLDBM/GDBM_File by PipTigger
in thread DBI vs MLDBM/GDBM_File, etc. by ZZamboni

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.