I'm not going to second (third? fifth? whatever... ;-}) the SQLite route. Well, not right away.

It seems that you have some sort of aversion to RDBMS's that I don't fully understand, but we can work with it. I suggest you convert your application to use DBI with DBD::CSV instead. DBD::CSV, contrary to what the name seems to imply, can use pipes as delimiters just fine, if you tell it to.

Once you've done that, you can fairly easily swap out the underlying datastore from flat, pipe-delimited files to SQLite or mySQL or DB2 or Oracle or ... whatever you want. Largely, this will mean just changing the DBI->connect line after moving your data over.

To be honest, DBD::CSV has been sufficient for me for a number of projects. But none of them have 15,000 records ;-) I went down this road thinking "this is a bunch of data - I should use some sort of query language to retrieve it, but I don't want to set up a database for only 200 records." So I used DBI to ensure I can switch out the underlying data structure at will. Or at least something close to "at will" - if someone writes a DBD::XML, I could use XML to store the data, but in the meantime, I have a lot of flexibility.

I would predict that once you convert this way, you'll be tempted to look at the real DBMS's, and, once you do, you will wonder why you took so long to switch. Your lookup times will drastically drop, and your manager will really like it. ;-)

In reply to Re: Large Constant Database in Text File by Tanktalus
in thread Large Constant Database in Text File by stephentyler

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.