I have spent the last two years building and maintaining a web application. As it gets more use, some weak spots are beginning to emerge. Clearly the largest one is where the Perl scripts (both CGI and administration) prod the database, asking "Got anything for me to do?" or "Hey! Give me a list of the documents according to these parameters!"

The symptom that seems to be most obvious is that things slow down as more documents (the system has to do with document processing) are added. I have one server with about ten systems running, most with about 100 documents -- it's absolutely fine. I have another server with 6000 documents -- it's better now than it was with 9000 documents, but it's still too slow.

Apart from wholesale changes like using POE, I'd like to be able to speed things up, or even prioritize things so that the CGI gets first dibs on the database, the "Hey!" jobs are next, and the administrative grunt scripts are at the bottom of the heap, grabbing spare cycles to do garbage collection whenever no one else is busy.

This morning as I sit, drink my coffee and ponder what approach might work, I'm thinking that some sort of cascading lock might work .. if the CGI sets a lock, then the "Hey!" task and the Janitor task will see that and stay away. Likewise, if the "Hey!" task sets a lock, the Janitor will stay away. If a higher rank task runs, it doesn't care if a lower rank task has set a lock and is running.

So, this isn't a fully cooked plan yet, but I thought I would throw it out there for feedback. Thanks monks!

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

In reply to Musings about a database ecology by talexb

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.