However, it is not the space requirements of the redundant data which worry me (with Terabyte disks within most people's reach that is no longer an issue), but rather keeping all those redundant data coordinated. It seems a big task which the RDBMS data-model has proven to solve.

Because the data used by OtoDB is effectively flat you can stripe it between servers without very much effort. You just have to query that same set of servers to ensure that you are getting all the data back.

Inserts work incrementally. The library makes an insert, and moves the pointer to the next server in the list and the next insert is done there. Somewhere, you must track the current insert server. In the RubberWiki demo I simply write the current server index to a text file. You could use a database table, or a dedicated server.

Queries, updates, and deletes are done against all servers in the list. So instead of sending a SQL command to one RDBMS, you send it to n servers. (Again, it's only this easy because the data structure is simplified. Fully relational data is much harder to try and distribute this way.) OtoDB does this sequentially, which will be a bottleneck at some point, I think. But my other intuition is that these three are parallelizable.

Of these, querying is the only one that requires a re-consolidation of the data. Update and delete just ask the data servers to perform some maintenance, and don't return anything to the user. When a query is run, it asks each server in the list to return the set of data generated by the SQL command (each server should return total_records/n records since incremental inserts should spread data evenly). In some cases the data will need to be reduced, and if ordering is requested, then a merge-order operation happens. These add additional work on the app side of things, but a pretty minimal amount from what I can see.

That's it, coordination in a nutshell.

A blog among millions.

If you're dealing with flat, denormalized data spread among several servers, then what are the advantages over other techniques?

You could work directly with fixed-width data files with fixed-width index files on a clustered file system. This solution lets the files system handle redundancy, distribution across multiple servers, and fault tolerance. The storage portion is already written, and it can be very efficient. You'd just need to write file handling, data locking, and search routines.

OpenLDAP allows you to write to one server (with failover to another) and query as many different servers as you want round-robin. Some other LDAP servers allow more than one server to accept writes at a time. If your data is more hierarchical than relational, then using a hierarchical database like a directory service makes sense. Every benchmark I've done or read elsewhere shows OpenLDAP having the lunch of RDBMS systems on write-seldom, read-often data.

If you're using relational databases, why are you querying servers in sequence to see which has the data? A good hashing algorithm for which DB server to query could cut down on quite a bit of traffic. Set up three different hash functions for three different data points in your data row. Hash against all three for each piece of data that comes in, and store to all three back-end servers that row maps to for each write. Then, you have three copies of everything, spread evenly among different servers (assuming good hash functions are selected). Then, you can hash against whichever portion you're querying against and get the data back out of just one server. Replicate the front-end, but don't bother replicating the back-end data stores because they're already storing in triplicate. If a data store server fails, you can reconstruct what it held from the front-end tables and the other data stores pretty easily, and in fact it'd be pretty simple to write a general-case program with DBI to do just that. As you have to scale up, you must adjust the hash functions to map to more back-end servers and prepopulate those servers with the appropriate data from the existing servers, but I don't see how to balance the storage load on new servers with your method at all other than pulling random rows across.