Working with the Replication Logs
BigMemory WAN Replication provides logging with messages that are easily parsable by third-party log watchers or scrapers. Logging includes:
Topology changes
Regularly print stats
Number of mutations.
Message Size
Data conflicts and repairs
Failover/recovery status/progress
Significant WAN replication log messages regard synchronization and incremental updates. The messages below provide important markers in the WAN replication logs.
Synchronization: Master Side
Log message issued when the Master cache is started synchronizing a Replica cache:
INFO [master-0] FullScanMasterSynchronizer - Master 'localhost:9001' initiated sync protocol for replica 'localhost:9003' and cache '__tc_clustered-ehcache|CacheManager|test-cache-1'
Log message issued when the Master cache is sending a synchronization batch to the Replica cache:
INFO [master-sync-13] FullScanMasterSynchronizer - Master 'localhost:9001' sent SYNC_UPDATE request with 7 events to replica 'localhost:9002' for cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache1'
Log message posting the total number of SYNC_UPDATEs that should be processed by the Replica:
INFO [New I/O worker #1] FullScanMasterSynchronizer - Total number of SYNC_UPDATE requests submitted to replica 'localhost:9002' is 184
Log message issued once the synchronization between Master and Replica has completed:
INFO [New I/O worker #4] FullScanMasterSynchronizer - Master 'localhost:9001' successfully synchronized replica 'localhost:9002' for cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache3'
Synchronization: Replica Side
Log message issued when the Replica cache has started synchronzing with the Master cache:
INFO [replica-sync-4] FullScanReplicaSynchronizer - Replica got SYNC_START request from master 'localhost:9001' for cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache3'
Log message issued when the Replica cache has processed a batch of synchronization updates from the Master cache:
INFO [replica-sync-2] FullScanReplicaSynchronizer - Replica got SYNC_UPDATE request from master 'localhost:9001' for cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache1' with 8 keys
Log message issued when the Replica cache has completed synchronization with Master cache, but the Replica is not yet activated:
INFO [replica-sync-5] FullScanReplicaSynchronizer - Replica got SYNC_END request from master 'localhost:9001' for cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache3'
Log message issued when the Replica cache is successfully activated after synchronization:
INFO [New I/O worker #1] ReplicaCache - Replica 'localhost:9002' activated cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache3' by request from master 'localhost:9001'
Incremental Updates
Log messages for incremental updates are for Bidirectional mode only. Note that watermarks are for batches of updates, which have been batched as part of the internal WAN process.
Log messages issued when the Master is receiving acknowledgements of a Replica successfully storing a batch of updates in the TSA:
INFO [master-0] UnitReplicator - Replica cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache1@localhost:9002' was at watermark '1131', is now at '1199'
INFO [master-0] MasterCache - Lowest watermark across all replicas for cache 'tc_clustered-ehcache|DEFAULT__|wan-test-cache1' is now '1199'
Log messages issued when a Replica has succesfully acklowledged the storage of the updates in the TSA to the Master cache:
INFO [New I/O worker #4] ReplicaCache - Replica 'tc_clustered-ehcache|DEFAULT__|wan-test-cache1@localhost:9002' successfully acknowledged watermark 1131 with master 'localhost:9001'
INFO [New I/O worker #4] ReplicaCache - Replica 'tc_clustered-ehcache|DEFAULT__|wan-test-cache1@localhost:9002' successfully acknowledged watermark 1199 with master 'localhost:9001'