Ok, so my perl-CGI database is simple. its not huge thank goodness but it is scalable. It is basically the database files (a database for each category, split into files by each 200 lines) and a database program. The database program goes though the database(s) line by line. with the index mode (defualt) it just goes though each 75 or 100 entries and splits them into fields and prints the stuff as nicely formatted html tables. nice little user list. since it goes line by line it saves memory.

Question 1) would it be faster to just print each of the database fields by fixed length or something like just finding a "-=-" after each one then putting them into an apporpriate list or whatever, or would it be just as fast to have individual delimiters for each field ("-=-", "=-=", "+-+", "-+-") and find fields by doing a sort of substr($line, index($line, "-=firstdelimiter=-"), index($line, "=-seconddelimiter-=") - index($line, "-=firstdelimiter=-") )

Next, it has a members database. every time a user clicks a form thingie the members page authentication program has to find the appropriate database file for their user id (very fast) and open it, find their data, and check their password against what the form gives us. every time. so how can we speed this process up? again, it goes line by line to conserve memory. since the files are not very large its not a lot of memory but i make my programs (and databases) so they are fast AND memory productive cuz who knows if my tiny little web host will be running the next yahoo right?

Question 2) how can we make this constant file checking fast and memory productive? and to make sure noone's pages get messed up, what kind of flocking system would we use to read/write to these files? could i just read and not worry about programs writing to it and just flock the writing process, or do i flock while writing AND reading from these files?

In reply to make perl text database fast? help! by psypete

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.