Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

DB Merge Replication

by trialmonkey (Hermit)
on Jun 30, 2004 at 14:28 UTC ( [id://370771]=perlquestion: print w/replies, xml ) Need Help??

trialmonkey has asked for the wisdom of the Perl Monks concerning the following question:

Prior to embarking on my quest to develop a database replication mechanism in Perl, I thought I'd tap the collective wisdom of my fellow monks. Basically, I need a reliable and manageable database replication mechanism for implementing a hub-and-spoke topology using merge replication. I'm currently using SQL Sever 2k replication, but find it has some holes and lacks any real management features.
  • The database is temporal in nature, and conflict resolution is not a big issue
  • Typically have about 100 sites (spokes) per hub
  • Data size is about 1MB per replication, sometimes over slow dial-up connections
  • Since opening firewalls requires work, HTTP tunneling might be worth considering
  • Currently all databases are SQL Server 2k, but I will need to include Oracle soon
  • Must be secure
  • Management of the overall replication process is important (IMO as important as the replication mechanism itself)

Due to the temporal nature of the database, it is possible to decouple replication from the database. That is, it would be possible to use a messaging scheme that would replicate the data and have a separate mechanism for handling database loading as well routing messages from the hub to the appropriate spokes.

Any design insights or pointers to useful modules is appreciated.

Replies are listed 'Best First'.
Re: DB Merge Replication
by mpeppler (Vicar) on Jun 30, 2004 at 15:39 UTC
    This is something that Sybase ASA combined with SQL Remote is really good at (which of course isn't really an answer that will be of any use to you...).

    Building this sort of system in perl, and have it be reliable is likely to be a pretty serious project. I'd start with building a messaging system of some sort to push the data from the spokes to the hub. The main issue that I can see is making sure that updates are applied correctly, and that failures are detected (and retried if necessary.)

    Michael

      Thanks. I haven't looked at the Sybase products in a few years. I'll revisit them.

      An open source solution would be ideal. The problem with the commercial products I've used is that while they're pretty solid, they all tend to have subtle bugs. With thousands of replicating nodes a few bugs translates into lots of questions regarding the state of the data. Because they're closed source, it's often difficult to understand what's really going on and get a real solution implemented.

      Building this sort of system in perl, and have it be reliable is likely to be a pretty serious project.

      Hence, I'm a SoPW and looking for some stable shoulders to stand on ;-)

Re: DB Merge Replication
by simonm (Vicar) on Jul 01, 2004 at 14:13 UTC
    I don't know of a cross-platform SQL-oriented toolkit for this. You could consider having each node write changes to a uniquely-named file, and then using existing file-replication techniques to make sure they end up where they're going...

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://370771]
Approved by sunadmn
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2024-03-29 00:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found