trialmonkey has asked for the wisdom of the Perl Monks concerning the following question:

Prior to embarking on my quest to develop a database replication mechanism in Perl, I thought I'd tap the collective wisdom of my fellow monks. Basically, I need a reliable and manageable database replication mechanism for implementing a hub-and-spoke topology using merge replication. I'm currently using SQL Sever 2k replication, but find it has some holes and lacks any real management features.

Due to the temporal nature of the database, it is possible to decouple replication from the database. That is, it would be possible to use a messaging scheme that would replicate the data and have a separate mechanism for handling database loading as well routing messages from the hub to the appropriate spokes.

Any design insights or pointers to useful modules is appreciated.

Replies are listed 'Best First'.
Re: DB Merge Replication
by mpeppler (Vicar) on Jun 30, 2004 at 15:39 UTC
    This is something that Sybase ASA combined with SQL Remote is really good at (which of course isn't really an answer that will be of any use to you...).

    Building this sort of system in perl, and have it be reliable is likely to be a pretty serious project. I'd start with building a messaging system of some sort to push the data from the spokes to the hub. The main issue that I can see is making sure that updates are applied correctly, and that failures are detected (and retried if necessary.)


      Thanks. I haven't looked at the Sybase products in a few years. I'll revisit them.

      An open source solution would be ideal. The problem with the commercial products I've used is that while they're pretty solid, they all tend to have subtle bugs. With thousands of replicating nodes a few bugs translates into lots of questions regarding the state of the data. Because they're closed source, it's often difficult to understand what's really going on and get a real solution implemented.

      Building this sort of system in perl, and have it be reliable is likely to be a pretty serious project.

      Hence, I'm a SoPW and looking for some stable shoulders to stand on ;-)

Re: DB Merge Replication
by simonm (Vicar) on Jul 01, 2004 at 14:13 UTC
    I don't know of a cross-platform SQL-oriented toolkit for this. You could consider having each node write changes to a uniquely-named file, and then using existing file-replication techniques to make sure they end up where they're going...