Because the data used by OtoDB is effectively flat you can stripe it between servers without very much effort. You just have to query that same set of servers to ensure that you are getting all the data back.

Inserts work incrementally. The library makes an insert, and moves the pointer to the next server in the list and the next insert is done there. Somewhere, you must track the current insert server. In the RubberWiki demo I simply write the current server index to a text file. You could use a database table, or a dedicated server.

Queries, updates, and deletes are done against all servers in the list. So instead of sending a SQL command to one RDBMS, you send it to n servers. (Again, it's only this easy because the data structure is simplified. Fully relational data is much harder to try and distribute this way.) OtoDB does this sequentially, which will be a bottleneck at some point, I think. But my other intuition is that these three are parallelizable.

Of these, querying is the only one that requires a re-consolidation of the data. Update and delete just ask the data servers to perform some maintenance, and don't return anything to the user. When a query is run, it asks each server in the list to return the set of data generated by the SQL command (each server should return total_records/n records since incremental inserts should spread data evenly). In some cases the data will need to be reduced, and if ordering is requested, then a merge-order operation happens. These add additional work on the app side of things, but a pretty minimal amount from what I can see.

That's it, coordination in a nutshell.

A blog among millions.

In reply to Re^4: RFC: OtoDB and rolling your own scalable datastore by arbingersys
in thread RFC: OtoDB and rolling your own scalable datastore by arbingersys

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.