I feel your pain. I, too, have become ridiculously adept at creating text-based databases (among other things, I write custom apps for integration with some "very legacy" products my company still uses for data collection...)

My advice (after just doing exactly what you're talking about) is that when you START on a project that you think text files will work just fine for, you need to consider the scope of the project. It's not really something you should be thinking about AFTER you've begun coding...I learned the hard way ;)

How much is the project going to grow (in terms of your procedures that work with the data)? If you have to do alot with the records, you're going to run into one of two situations - either you'll take a performance hit opening, iterating through, and closing the files everytime you need to search for some data OR you'll take a performance hit by loading it all into a hash in memory and searching through the index...the hash cuts down on the opening/while-ing/closing time, improves your appending/inserting/modifying procedures, but sucks up overhead. (And there's only so much that compressing the string before it ends up in the hash can do for you...it's not really worth it unless your strings are super-long.)

Also consider how much the data is going to grow physcially. Even if you're not doing anything truly wacky with appending/modifying/inserting records, if the # of records is going to grow, you've got management issues...hashed data or not, it can get to be very tedious.

Finally, is your project likely to become part of a larger system? If it's stand-alone and flat files still seem like a good idea, go for it...BUT if you're likely to be integrating with other reporting/content-generating software, go with the database. Just about everything can do an ODBC connection.

I agree, to a point, with the posters that suggest developing your relat. db skills. That's very cool and all, but there are some situations where you may get better performance out of a flat file...that bears consideration, as well. Nothing ticks me off more than going to a website that is slow because it UNNECESSARILY generates some of its content from a database.

Without knowing too much about your project, I would suggest developing those relat. db skills (look into PHP for talking with them!) for future projects and leave the current project alone. If it works and is not going to grow its database to a tremendous size, you should probably be looking ahead at more interesting projects rather than looking back to fine-tune this one.

In reply to Re: Text Database by clearcache
in thread Text Database by EyesOnly

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.