This document is organized as follows:
This section describes the various statistics (or metrics) sent from the Event Replicator Server and made available to the Replication Monitoring subsystem. Snapshots of these statistics may also be seen via the DRPLSTAT operator command, or through the Adabas Online System (AOS). The format of these statistics may differ slightly between the different facilities.
While each metric reflects specific information about a certain aspect of the replication process, combinations of some metrics could make the statistical information and trends more meaningful; for example:
If monitoring the SLOG status, a change in items logged and de-logged, to the SLOG file, may reflect the load on the Replication system. For instance, if the number of de-logged items increases at a faster rate than new items are logged, then the Event Replicator Server through-put could be deemed to be optimistic. However, if the logged items are increasing faster than items are being de-logged, it may imply that the Event Replicator Server is receiving a higher load than normal or a bottleneck may be occurring. This may be also verified by looking at the Input Transactions.
While a single snapshot may seem concerning, it may merely mean that a large transaction or batch job, doing heavy updates, may be in progress. Further monitoring of the metrics would verify this trend more accurately.
This section covers the following statistical categories originating from the Event Replicator Server:
Input transactions : 25 Pending input transactions : 0 Output items : 470 Pending output items : 0 Output messages : 498 Bytes sent : 1,601,617 Input messages : 29,364 Bytes received : 26,374,377 Input commits : 29,357 Input backouts : 0 Items de-logged : 41 Items logged : 41 Items on SLOG : 0 Database-related input transactions: De-logged : 0 Logged : 0 On SLOG : 0 FCBs on SLOG : 0
The global statistics section shows statistics collected for the entire Event Replicator Server session.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticGlobal | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| Input transactions | Shows the number of fully processed input replicated transactions. | inputTransactions |
| Pending input transactions | Shows the number of in-flight input transactions. | pendinInputTransactions |
| Output items | Shows the total number of output transactions and events (i.e. URBS) for all destinations. | |
| Pending output items | Shows the total number of in-flight output transactions and events for all destinations. | |
| Output messages | Shows the total number of output messages sent for all destinations. | totalMessagesAllDests |
| Bytes sent | Shows the total number of bytes in the output messages. | totalBytesAllDests |
| Input messages | Shows the total number of input messages received for all input queues. | totalMessagesAllQueues |
| Bytes received | Shows the total number of bytes in the input messages. | totalBytesAllQueues |
| Input commits | Shows the total number of commits issued for all input queues. | totalCommitsAllQueues |
| Input backouts | Shows the total number of backouts issued for all input queues. | totalBackoutsAllQueues |
| Items de-logged | Shows the total number of transactions and events read from the SLOG system file for all destinations including database-related input transactions. | totalItemsDeloggedSLOG |
| Items logged | Shows the total number of transactions and events written to the SLOG system file for all destinations including database-related input transactions. | totalItemsLoggedSLOG |
| Items on SLOG | Shows the total number of transactions and events currently residing on the SLOG system file for all destinations including database-related input transactions. | totalItemsOnSLOG |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength |
The Database-related input transactions section has the following statistics:
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| De-logged | Shows the number of database-related input transactions read from the SLOG system file. | dbTransDeloggedSLOG |
| Logged | Shows the number of database-related input transactions written to the SLOG system file. | dbTransLoggedSLOG |
| On SLOG | Shows the number of database-related input transactions currently residing on the SLOG system file. | dbTransOnSLOG |
| FCBs on SLOG | Shows the number of Adabas File Control Blocks (FCBs) currently residing on the SLOG system file. | dbFCBFDTTransOnSLOG |
SUB1 Input items : 24 Output items : 26 C5 data : 0 Data lost : 1 Initial-state completed : 1 Initial-state data : 23 Security functions : 1 User transactions : 0 Utility functions : 0 Item time at source : 2017-10-17 18:45:41 Item processed : 2017-10-17 18:45:41
The subscription statistics section is repeated for each subscription defined to the Event Replicator Server.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticSubscription | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| SUB1 (sample) | Subscription name | subscriptionName |
| Input items | Shows the number of input transactions processed for this subscription. | |
| Output items | Shows the total number of output transactions and events (i.e. URBS) for this subscription. | subscriptionOutputItems |
| C5 data | Shows the number of C5 events created for this subscription. | subscriptionC5DataTransactions |
| Data lost | Shows the number of data lost events created for this subscription. | subscriptionDataLostTransactions |
| Initial-state completed | Shows the number of initial-state completed events created for this subscription. | subscriptionInitialStateCompletions |
| Initial-state data | Shows the number of initial-state transactions processed for this subscription. | subscriptionInitialStateDataTransactions |
| Security functions | Shows the number of security function events created for this subscription. | subscriptionSecurityFunctions |
| User transactions | Shows the number of replicated transactions processed for this subscription. | subscriptionUserDataTransactions |
| Utility functions | Shows the number of Adabas utility-related events created for this subscription. | subscriptionUtilityFunctions |
| Item time at source | Is the commit time of the last processed transaction or event for this subscription. | subscriptionLastItemCommitTime |
| Item processed | Is the end of subscription processing time for the last processed transaction or event for this subscription. | subscriptionLastItemProcessedTime |
| subscriptionLastItemProcessedTime minus subscriptionLastItemCommitTime | subscriptionLastItemDurationTime | |
| subscriptionOutputItems divided by intervalLength | subscriptionItemsPerSecond | |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength |
DBID 28857 File 1 Deletes : 0 Filtered out : 1 Initial-state : 0 Inserts : 1,106 Updates : 2
The subscription file statistics section is repeated for each database ID (SFDBID) and file (SFILE) combination defined to an individual subscription.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticSubscriptionFile | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| Subscription name | subscriptionName | |
| DBID 28857 (sample) | Is the database ID in the above sample. | subscriptionFileDBID |
| File 1 (sample) | Is the file ID in the above sample. | subscriptionFileFNR |
| Deletes | Shows the number of deletes processed for the file. | subscriptionFileDeletes |
| Filtered out | Shows the number of records which were excluded from replication processing by transaction filtering for the file. | |
| Initial-state | Shows the number of initial-state transactions processed for the file. | subscriptionFileInitialStateRecords |
| Inserts | Shows the number of inserts processed for the file. | subscriptionFileInserts |
| Updates | Shows the number of updates processed for the file. | subscriptionFileUpdates |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength |
Deleted subscriptions Output items : 1
| Label | Value Description |
|---|---|
| Output items | Shows the total number of output transactions and events (i.e. URBS) for all deleted subscriptions. |
DEST1 Output items : 137 Pending items : 0 Item time at source : 2017-10-17 18:59:31 Messages : 165 Bytes : 1,401,542 Commits : 128 Commit time at destination : 2017-10-17 18:59:31 Pending messages : 1 Pending bytes : 110 De-logged : 11 Logged : 11 On SLOG : 0
The destination statistics section is repeated for each destination defined to the Event Replicator Server.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticDestinationBrokerMQNull | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| DEST1 (sample) | Is the destination name in this example. | destinationName |
| Output items | Shows the number of output transactions and events (i.e. URBS) for the destination. | destOutputItems |
| Pending items | Shows the number of in-flight output transactions and events for the destination. | destPendingItems |
| Item time at source | Is the commit time of last processed transaction or event for the destination. | destItemTimeAtSource |
| Messages | Shows the number of output messages sent for the destination. | destMessages |
| Bytes | Shows the number of bytes in the output messages. | destBytes |
| Commits | Shows the number of messaging system commits for the destination. | destCommits |
| Commit time at destination | Is the time of the last commit issued to the messaging system for the destination. | destCommitTimeAtDestination |
| Pending messages | Shows the number of messages that have been written but not yet committed to the messaging system for the destination. | destPendingMessages |
| Pending bytes | Shows the number of bytes for the pending messages. | destPendingBytes |
| De-logged | Shows the number of transactions and events read from the SLOG system file for the destination. | destItemsDeloggedSLOG |
| Logged | Shows the number of transactions and events written to the SLOG system file for the destination. | destItemsLoggedSLOG |
| On SLOG | Shows the number of transactions and events currently residing on the SLOG system file for the destination. | destItemsOnSLOG |
| destLastMessagingCommitTime minus destLastTransCommitTime | destLastItemDurationTime | |
| destReplicatedTrans divided by intervalLength | destItemsPerSecond | |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength |
ADAFI15 Output items : 54,710 Pending items : 0 Item time at source : 2017-10-17 19:12:36 BT : 0 ET : 21,500 ET time at destination : 2017-10-17 19:14:07 Input: DBID 28857 File 15 Target: DBID 28858 File 15 A1 : 6,600 E1 : 7,450 HI : 0 N1 : 1 N2 : 7,450 Security : 1 S4 : 1 UTI : 0
The Adabas destination statistics section is repeated for each Adabas destination defined to the Event Replicator Server.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticDestinationAdabasFile | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| ADAFI15 (sample) | Is the Adabas destination name in the above sample. | destinationName |
| Output items | Shows the number of output transactions and events (i.e. URBS) for the destination. | destOutputItems |
| Pending items | Shows the number of in-flight output transactions and events for the destination. | destPendingItems |
| Item time at source | Is the commit time of last processed transaction or event for the destination. | destItemTimeAtSource |
| BT | Shows the number of BT (backout transaction) commands successfully issued for the destination. | destBTCommands |
| ET | Shows the number of ET (end transaction) commands successfully issued for the destination. | destETCommands |
| ET time at destination | Is the time of the last successful ET command that was issued for the destination. | destETTimeAtDestination |
| Items de-logged from the SLOG system file. | destItemsDeloggedSLOG | |
| Items logged to the SLOG system file. | destItemsLoggedSLOG | |
| Items on the SLOG system file. | destItemsOnSLOG | |
| destLastETTime minus destLastTransCommitTime | destLastTransactionDurationTime | |
| destReplicatedTrans divided by intervalLength | destTransactionsPerSecond | |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength | |
|
Input: DBID nnnnn File xx Target: DBID nnnnn File xx |
Is the input database/file and the target database/file combination. |
destInputDBID destInputFNR destTargetDBID destTargetFNR |
| destNumberErrors | ||
| A1 | Shows the number of A1 (update) commands issued for the destination. | destUpdateCommands |
| E1 | Shows the number of E1 (delete) commands issued for the destination. | destDeleteCommands |
| HI | Shows the number of HI (hold record) commands issued for the destination. | destHoldCommands |
| N1 | Shows the number of N1 (add) commands issued for the destination. | destInsertCommands |
| N2 | Shows the number of N2 (add) commands issued for the destination. | destInsertISNCommands |
| Security | Shows the number of Adabas security operations replicated to the target database/file. | destSecurityFunctions |
| S4 | Shows the number of S4 (find) commands issued for the destination. | destFindCommands |
| UTI | Shows the number of Adabas utility operations replicated to the target database/file. | destUtilityFunctions |
| Replication method ('ISN ' / 'KEY ') | destReplicationMethod |
Deleted destinations Output items : 1 Messages : 1 Bytes : 110 Commits : 1 De-logged : 1 Logged : 1
| Label | Value Description |
|---|---|
| Output items | Shows the number of output transactions and events (i.e. URBS) for all deleted destinations. |
| Messages | Shows the number of output messages sent for all deleted destinations. |
| Bytes | Shows the number of bytes in the output messages. |
| Commits | Shows the number of messaging system commits for all deleted destinations. |
| De-logged | Shows the number of transactions and events read from the SLOG system file for all deleted destinations. |
| Logged | Shows the number of transactions and events written to the SLOG system file for all deleted destinations. |
Output subtask items:
Main : 0
1 : 268
2 : 101
3 : 101
Output subtask message totals:
Main : 0
1 : 1
2 : 1
3 : 1
The output subtask items detail line is repeated for each output subtask defined to the Event Replicator Server. Each detail line shows the number of output transactions and events (i.e. URBS) processed for the subtask.
The output subtask message totals detail line is repeated for each output subtask defined to the Event Replicator Server. Each detail line shows the number of messages processed for the subtask.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticOutputTask | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| Main (sample) | taskNumber (0 = main task) | taskNumber |
| items | Shows the number of output transactions and events (i.e. URBS) processed for the subtask. | items |
| message totals | Shows the number of messages processed for the subtask. | messagesSent |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength |
INPUTQ1 Input items : 54,721 Pending items : 0 Item time at source : 2017-10-17 19:10:45 Messages : 29,364 Bytes : 26,374,377 Pending messages : 1 Bytes : 110 Commits : 29,357 Commit time at input queue : 2017-10-17 19:15:45 Backouts : 0
The input queue statistics section is repeated for each input queue defined to the Event Replicator Server.
| Label | Value Description | Elasticsearch Field |
|---|---|---|
| ReptorStatisticInputQueue | type | |
| URBS reptor status ("RSTA") | reptorStatus | |
| Target ID of originating Event Replicator Server | targetID | |
| Nucleus ID of originating Event Replicator Server | nucleusID | |
| INPUTQ1 (sample) | Is the input queue name in this example. | queueName |
| Input items | Shows the number of input transactions and events (i.e. URBS) for the input queue. | |
| Pending items | Shows the number of in-flight input transactions and events (i.e. URBS) for the input queue. | |
| Item time at source | Is the time of the last input transaction or event processed for the input queue. | |
| Messages | Shows the number of messages read and committed for the input queue. | queueMessagesReceived |
| Bytes | Shows the number of bytes in the above messages. | queueBytesReceived |
| Pending messages | Shows the number of messages read from the input queue but not yet committed. | queuePendingMessages |
| Bytes | Shows the number of bytes in the above messages. | queuePendingBytes |
| Commits | Shows the number of commits issued for the input queue. | queueMessagingCommits |
| Commit time at input queue | Is the time the last commit was issued for the input queue. | |
| Backouts | Shows the number of back outs issued for the input queue. | queueMessagingBackouts |
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp | |
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd | |
| interval length in milliseconds timestampEnd minus timestamp | intervalLength |
Deleted input queues Input items : 1 Messages : 1 Bytes : 110 Commits : 1 Backouts : 0
| Label | Value Description |
|---|---|
| Input items | Shows the number of input transactions and events (i.e. URBS) for all deleted input queues. |
| Messages | Shows the number of messages read and committed for all deleted input queues. |
| Bytes | Shows the number of bytes in the above messages. |
| Commits | Shows the number of commits issued for all deleted input queues. |
| Backouts | Shows the number of back outs issued for all deleted input queues. |
This section describes the various statistics (or metrics) sent from the Event Replicator Target Adapter and made available to the Replication Monitoring subsystem. The Event Replicator Target Adapter has always collected statistics and made them available for display using Event Replicator Target Adapter Administration. These statistics are described in the Event Replicator Target Adapter documentation at Reviewing Event Replicator Target Adapter Statistics.
The statistics collected and displayed by the Event Replicator Target Adapter and its Administration tool are now also sent to Elasticsearch and made available to Kibana dashboards and visualizations as part of Replication Monitoring. The "Category Type" in the Event Replicator Target Adapter statistics documentation maps directly to the Elasticsearch fields "type" and "id". Some examples of this mapping are shown in the table below:
| Category Type | Elasticsearch “type” value | Elasticsearch “id” value |
|---|---|---|
| /engine/… | engine | engine id e.g. "Engine-0" |
| /source/sourcename | source | sourcename e.g. "HOST3-S1" |
| /jdbc/targetname | jdbc | targetname e.g. "host05 SQL Server" |
| /subscription/subname | subscription | subname e.g. "EAPSUB3" |
| /table/tablename | table | tablename e.g. "Employees" |
| /heartbeat/destname | heartbeat | destname e.g. "DEST2" |
| /heartbeat_etb/destname | heartbeat_etb | destname e.g. "DEST2" |
| /heartbeat_processing/destname | heartbeat_processing | destname e.g. "DEST2" |
| /heartbeat_slog/destname | heartbeat_slog | destname e.g. "DEST2" |
| /latency/source.target.subname | latency | source.target.subname e.g. "HOST3-S1.host05 SQL Server.EAPSUB3" |
Please refer to Reviewing Event Replicator Target Adapter Statistics for further details.
The statistics reported by the Event Replicator Target Adapter include a Heartbeat functionality that merits some additional description. A heartbeat is a special short message generated and sent down the pipe through the messaging system and into the Target Adapter. The purpose of this message is to ensure that all pieces of the replication system are working properly at a regular interval. The Target Adapter recognizes these heartbeat messages and reports statistics about the flow timing as part of Replication Monitoring.
The full set of statistics that includes all properties of the heartbeat is written at the conclusion of all heartbeat processing as an Elasticsearch data point with field "type" set to "heartbeat". Additional data points are written containing certain sub-types that comprise the full heartbeat. These are written with field "type" set to "heartbeat_etb", "heartbeat_processing", and "heartbeat_slog". All data points contain a "timestamp" field with different values appropriate for the sub-type. The final full heartbeat data point is written with "timestamp" set to the final time when all heartbeat statistics have been fully recognized and processed. Some important points in the heartbeat timeline include:
| Timestamp Name | Timestamp Description |
|---|---|
| creationTime | Time the URBS heartbeat message was created on the mainframe by Event Replicator for Adabas. |
| readFromSlog | Time the URBS heartbeat message was read from the SLOG file on the mainframe by Event Replicator for Adabas. This timestamp will only have a value if the mainframe processing load was sufficiently high to cause the message to be written to the SLOG file for later processing. |
| infoTime | Time when Unit-of-Work status was obtained from EntireX Broker by Event Replicator Target Adapter. |
| artTime | Time when the heartbeat message is fully received and recognized by Event Replicator Target Adapter. |
The table below shows all heartbeat "type" values with the "timestamp" value used and the list of Elasticsearch fields included in the data points for that sub-type.
| Elasticsearch “type” value | Elasticsearch “timestamp” value | Elasticsearch fields |
|---|---|---|
| heartbeat_etb | infoTime |
maxBytes |
| heartbeat_processing | infoTime | processingTime |
| heartbeat_slog | readFromSlog | slogTime |
| heartbeat | artTime | Complete field list described at Reviewing Event Replicator Target Adapter Statistics in table entry Heartbeat. |
Please consult Reviewing Event Replicator Target Adapter Statistics for more details.