InstantdbSync replicates transactions or Events from a source database to a target database.  Source tables can be replicated with the same schema as the source or transformations can be performed using column mapping capabilities found in the Replication Administrator.  This process of updating the target database in real-time is referred to as Event Replication.  The components that make up InstantdbSync are referred to as the Replication Components.

 

There are three major stages of replication.  The deploy process, the initial state process and the replicating process.

1. The deploy process:
   Once the desired source and target tables have been identified and any column mapping done, the replications need to be deployed to the replication controller.  During this process, the Controller analyzes the replications and notifies the Event Producer (EP) of which tables are being replicated from the source.  It also notifies the Event Consumer (EC) of which target tables the data is destined for.  Once the EP and EC have processed the information that was deployed, they will send a status message back to the Controller to indicate success or failure.  If the deploy was successful, the Controller will initiate the Initial State process.

2. The Initial State process:
   Before replication can begin, the target table must first be in sync with the source table.  To achieve this, any previously existing data in the target must be deleted and then a copy of all the rows from the source table must be moved t the target table.  This is known as the Initial State process.  While an Initial State is running, any changes made to the source database will be held in the message queue and will be processed after a successful completion of the Initial State.  Note, if deploy is adding an additional replication or replications to a list of previously deployed replications, only the newly added replications will go through the initial state process.

3. Replication:
   After the Initial State is finished, the systems enters a state where it is monitoring the source database transaction logs and replicating transactions as they occur.

 

Replication Overview from the "front end" perspective

To replicate data using the InstantdbSync:

1. Verify that you already have a CONNX Data Dictionary (CDD) containing the file definitions for the source files.

2. Add a link to the desired target database to the data dictionary.

3. Start the InstantdbSync Replicator Administrator and select the CDD.

The first time you open the CDD you will be prompted to enter the location of the replication server components (Controller), and specify any port overrides for communicating to the message queue.

4. Select the source tables to replicate, and the target database.

• InstantdbSync assumes a one-for-one column mapping between the source and the target tables.  Use the InstantdbSync Replication Administrator to change the source column to target column mapping.

• You can map SQL expressions to target columns. This provides a powerful transformation capability.

5. Add replication filters by using built-in features of the CONNX Data Dictionary Manager. Using the Data Dictionary Manager, a SQL based filter can be applied to any table in the "SQL View Clause" field. For more information on using the SQL View Clause, please see the CONNX User Reference Guide.

6.  Press Deploy, and real-time event replication begins.

 

Replication Overview from the "back end" perspective

When you deploy a replication:

1. The InstantdbSync Replication Administrator uses the "front end" information to create a new replication CDD and sends it to the Controller.

2. The Controller pauses any active Event Consumers (EC).

3. The Controller deploys the new replication CDD in the proper location.

4. After reading the instructions from the new replication CDD, new event filters are sent to the Event Producer (EP) through the Message Queue (MQ). The EP captures only events for the replicated tables, and ignores all other events.

5. The Controller starts up one or more EP and EC engines depending on the work to be done. An EP and EC is started for every source DBID/target database combination in the new replication CDD.  If there were any previously active ECs that were paused in step 2, they will be started again at this point.

6. The Controller determines whether an "initial state" is required based on persistent state information stored in a memory mapped file. If an initial state is required, the Controller instructs the EC to start the initial state processing.

Initial states are performed by reading the source data using CONNX, and inserting it into the target database using a standard INSERT/SELECT SQL statement. During initial state processing, changes to the source tables are kept in the Message Queue for future playback after initial state processing is complete.

 

Note: If a new table was added to an existing EP/EC combination of replications, the new table will have an Initial state performed but the existing tables will not.  The replications for the existing tables will be paused during the initial state process.  As with the new table(s), any transactions that occur against the existing replications will be queued until the initial state process finishes.

7. After any initial state processing, the EP captures changes to the selected tables and places them on the MQ. The EC reads the MQ and uses CONNX to place changes in the target database. This continues in real time until replication is stopped.