DB Merge Replication

trialmonkey has asked for the wisdom of the Perl Monks concerning the following question:

Prior to embarking on my quest to develop a database replication mechanism in Perl, I thought I'd tap the collective wisdom of my fellow monks. Basically, I need a reliable and manageable database replication mechanism for implementing a hub-and-spoke topology using merge replication. I'm currently using SQL Sever 2k replication, but find it has some holes and lacks any real management features.

The database is temporal in nature, and conflict resolution is not a big issue
Typically have about 100 sites (spokes) per hub
Data size is about 1MB per replication, sometimes over slow dial-up connections
Since opening firewalls requires work, HTTP tunneling might be worth considering
Currently all databases are SQL Server 2k, but I will need to include Oracle soon
Must be secure
Management of the overall replication process is important (IMO as important as the replication mechanism itself)

Due to the temporal nature of the database, it is possible to decouple replication from the database. That is, it would be possible to use a messaging scheme that would replicate the data and have a separate mechanism for handling database loading as well routing messages from the hub to the appropriate spokes.

Any design insights or pointers to useful modules is appreciated.

Comment on DB Merge Replication

Replies are listed 'Best First'.
Re: DB Merge Replication by mpeppler (Vicar) on Jun 30, 2004 at 15:39 UTC
This is something that Sybase ASA combined with SQL Remote is really good at (which of course isn't really an answer that will be of any use to you...). Building this sort of system in perl, and have it be reliable is likely to be a pretty serious project. I'd start with building a messaging system of some sort to push the data from the spokes to the hub. The main issue that I can see is making sure that updates are applied correctly, and that failures are detected (and retried if necessary.) Michael	[reply]
Re^2: DB Merge Replication by trialmonkey (Hermit) on Jul 01, 2004 at 01:36 UTC
Thanks. I haven't looked at the Sybase products in a few years. I'll revisit them. An open source solution would be ideal. The problem with the commercial products I've used is that while they're pretty solid, they all tend to have subtle bugs. With thousands of replicating nodes a few bugs translates into lots of questions regarding the state of the data. Because they're closed source, it's often difficult to understand what's really going on and get a real solution implemented. Building this sort of system in perl, and have it be reliable is likely to be a pretty serious project. Hence, I'm a SoPW and looking for some stable shoulders to stand on ;-)	[reply]
Re: DB Merge Replication by simonm (Vicar) on Jul 01, 2004 at 14:13 UTC
I don't know of a cross-platform SQL-oriented toolkit for this. You could consider having each node write changes to a uniquely-named file, and then using existing file-replication techniques to make sure they end up where they're going...	[reply]