Hi folks,

I have a problem. In a nutshell I have a homegrown database application that uses a DBM tie to provide data access. Concurrency is handled by flock, only providing access to a single writer, or many readers at any time.

As the number of people using this application grows, it's not scaling too well. For obvious reasons. We need row level locking, or SQL or something.

To give you an idea of the seriousness, some write operations are taking upward of 30 seconds. That's 30 seconds where no one can use the entire system. That's bad for an interactive app, in anyones language :-)

Yes SQL is on the cards, but it will require a major rewrite, which we aren't really ready for yet. Half-assed SQL conversions (basically a DBM file in SQL) are also an option, but require some API changes.

I thought this morning of a change which would require a very small amount of changes to the API, with plenty of potential gains. It might also be generally useful for the perl community at large.

My idea is to provide a wrapper around a DBM tie that would provide the following extra functionality:

  1. Delayed writes
  2. Read caching
So write operations would always succeed, instantly. Read operations on common keys would be cached in memory (this may not be a win, I imagine the OS caching probably already gets you this for free with a DBM tie).

The delayed writes is a huge win though. If write operations can be deferred until the system is not busy, or until a certain amount of time has elapsed, interactive performance will improve markedly.

I imagine that the simplest implementation would be a module which talks to a named pipe or socket, which has a seperate script running on the other side. A nice failsafe could be that the client side of the thing could fall back to normal tie behaviour if the server side can't be contacted.

Firstly, am I insane to think this could be done well in pure perl?

Secondly, has anyone already done this? I can't see anything on CPAN, but maybe I'm not looking in the right places.

Lastly, I really, really need to be able to preserve my current behaviour where database operations are simply reads or writes to hash keys. I'm moving away from that but it's a slow process. Any solutions which change the way I'd read and write my data significantly are really useless to me at this stage.

Thanks.

In reply to Caching DBM hash tie by Tardis

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.