Replication Monitoring Metrics

#### Search Results

Search

# Replication Monitoring Metrics

This document is organized as follows:

- Event Replicator Server Statistics

- Event Replicator Target Adapter Statistics

- Heartbeat

## Event Replicator Server Statistics

This section describes the various statistics (or metrics) sent from the Event Replicator Server and made available to the Replication Monitoring subsystem. Snapshots of these statistics may also be seen via the DRPLSTAT operator command, or through the Adabas Online System (AOS). The format of these statistics may differ slightly between the different facilities.

While each metric reflects specific information about a certain aspect of the replication process, combinations of some metrics could make the statistical information and trends more meaningful; for example:

If monitoring the SLOG status, a change in items logged and de-logged, to the SLOG file, may reflect the load on the Replication system. For instance, if the number of de-logged items increases at a faster rate than new items are logged, then the Event Replicator Server through-put could be deemed to be optimistic. However, if the logged items are increasing faster than items are being de-logged, it may imply that the Event Replicator Server is receiving a higher load than normal or a bottleneck may be occurring. This may be also verified by looking at the Input Transactions.

While a single snapshot may seem concerning, it may merely mean that a large transaction or batch job, doing heavy updates, may be in progress. Further monitoring of the metrics would verify this trend more accurately.

This section covers the following statistical categories originating from the Event Replicator Server:

- Global Statistics

- Subscription Statistics

- Subscription File Statistics

- Deleted Subscription Statistics

- Destination Statistics

- Adabas Destination Statistics

- Deleted Destination Statistics

- Subtask Statistics

- Input Queue Statistics

- Deleted Input Queue Statistics

### Global Statistics

#### Sample output

    Input transactions             :               25
    Pending input transactions     :                0
    Output items                   :              470
    Pending output items           :                0
    Output messages                :              498
     Bytes sent                    :        1,601,617
    Input messages                 :           29,364
     Bytes received                :       26,374,377
    Input commits                  :           29,357
    Input backouts                 :                0
    Items de-logged                :               41
    Items logged                   :               41
    Items on SLOG                  :                0

    Database-related input transactions:
      De-logged                    :                0
      Logged                       :                0
      On SLOG                      :                0
      FCBs on SLOG                 :                0

#### Labels and values shown for Global Statistics

The global statistics section shows statistics collected for the entire Event Replicator Server session.

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticGlobal | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
Input transactions | Shows the number of fully processed input replicated transactions. | inputTransactions
Pending input transactions | Shows the number of in-flight input transactions. | pendinInputTransactions
Output items | Shows the total number of output transactions and events (i.e. URBS) for all destinations. |
Pending output items | Shows the total number of in-flight output transactions and events for all destinations. |
Output messages | Shows the total number of output messages sent for all destinations. | totalMessagesAllDests
Bytes sent | Shows the total number of bytes in the output messages. | totalBytesAllDests
Input messages | Shows the total number of input messages received for all input queues. | totalMessagesAllQueues
Bytes received | Shows the total number of bytes in the input messages. | totalBytesAllQueues
Input commits | Shows the total number of commits issued for all input queues. | totalCommitsAllQueues
Input backouts | Shows the total number of backouts issued for all input queues. | totalBackoutsAllQueues
Items de-logged | Shows the total number of transactions and events read from the SLOG system file for all destinations including database-related input transactions. | totalItemsDeloggedSLOG
Items logged | Shows the total number of transactions and events written to the SLOG system file for all destinations including database-related input transactions. | totalItemsLoggedSLOG
Items on SLOG | Shows the total number of transactions and events currently residing on the SLOG system file for all destinations including database-related input transactions. | totalItemsOnSLOG
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength

#### Labels and values shown for Database-related input transactions

The Database-related input transactions section has the following statistics:

Label | Value Description | Elasticsearch Field
--- | --- | ---
De-logged | Shows the number of database-related input transactions read from the SLOG system file. | dbTransDeloggedSLOG
Logged | Shows the number of database-related input transactions written to the SLOG system file. | dbTransLoggedSLOG
On SLOG | Shows the number of database-related input transactions currently residing on the SLOG system file. | dbTransOnSLOG
FCBs on SLOG | Shows the number of Adabas File Control Blocks (FCBs) currently residing on the SLOG system file. | dbFCBFDTTransOnSLOG

### Subscription Statistics

#### Sample output

    SUB1
      Input items                  :               24
      Output items                 :               26
       C5 data                     :                0
       Data lost                   :                1
       Initial-state completed     :                1
       Initial-state data          :               23
       Security functions          :                1
       User transactions           :                0
       Utility functions           :                0
      Item time at source          : 2017-10-17 18:45:41
      Item processed               : 2017-10-17 18:45:41

#### Labels and values shown for subscription statistics

The subscription statistics section is repeated for each subscription defined to the Event Replicator Server.

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticSubscription | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
SUB1 (sample) | Subscription name | subscriptionName
Input items | Shows the number of input transactions processed for this subscription. |
Output items | Shows the total number of output transactions and events (i.e. URBS) for this subscription. | subscriptionOutputItems
C5 data | Shows the number of C5 events created for this subscription. | subscriptionC5DataTransactions
Data lost | Shows the number of data lost events created for this subscription. | subscriptionDataLostTransactions
Initial-state completed | Shows the number of initial-state completed events created for this subscription. | subscriptionInitialStateCompletions
Initial-state data | Shows the number of initial-state transactions processed for this subscription. | subscriptionInitialStateDataTransactions
Security functions | Shows the number of security function events created for this subscription. | subscriptionSecurityFunctions
User transactions | Shows the number of replicated transactions processed for this subscription. | subscriptionUserDataTransactions
Utility functions | Shows the number of Adabas utility-related events created for this subscription. | subscriptionUtilityFunctions
Item time at source | Is the commit time of the last processed transaction or event for this subscription. | subscriptionLastItemCommitTime
Item processed | Is the end of subscription processing time for the last processed transaction or event for this subscription. | subscriptionLastItemProcessedTime
| subscriptionLastItemProcessedTime minus subscriptionLastItemCommitTime | subscriptionLastItemDurationTime
| subscriptionOutputItems divided by intervalLength | subscriptionItemsPerSecond
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength

### Subscription File Statistics

#### Sample output

    DBID 28857 File     1
       Deletes                     :                0
       Filtered out                :                1
       Initial-state               :                0
       Inserts                     :            1,106
       Updates                     :                2

#### Labels and values shown for subscription file statistics

The subscription file statistics section is repeated for each database ID (SFDBID) and file (SFILE) combination defined to an individual subscription.

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticSubscriptionFile | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
| Subscription name | subscriptionName
DBID 28857 (sample) | Is the database ID in the above sample. | subscriptionFileDBID
File 1 (sample) | Is the file ID in the above sample. | subscriptionFileFNR
Deletes | Shows the number of deletes processed for the file. | subscriptionFileDeletes
Filtered out | Shows the number of records which were excluded from replication processing by transaction filtering for the file. |
Initial-state | Shows the number of initial-state transactions processed for the file. | subscriptionFileInitialStateRecords
Inserts | Shows the number of inserts processed for the file. | subscriptionFileInserts
Updates | Shows the number of updates processed for the file. | subscriptionFileUpdates
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength

### Deleted Subscription Statistics

#### Sample output

    Deleted subscriptions
      Output items                 :                1

#### Labels and values shown for deleted subscription statistics

Label | Value Description
--- | ---
Output items | Shows the total number of output transactions and events (i.e. URBS) for all deleted subscriptions.

### Destination Statistics

#### Sample output

    DEST1
      Output items                 :              137
       Pending items               :                0
      Item time at source          : 2017-10-17 18:59:31
      Messages                     :              165
       Bytes                       :        1,401,542
      Commits                      :              128
      Commit time at destination   : 2017-10-17 18:59:31
      Pending messages             :                1
       Pending bytes               :              110
      De-logged                    :               11
      Logged                       :               11
      On SLOG                      :                0

#### Labels and values shown for destination statistics

The destination statistics section is repeated for each destination defined to the Event Replicator Server.

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticDestinationBrokerMQNull | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
DEST1 (sample) | Is the destination name in this example. | destinationName
Output items | Shows the number of output transactions and events (i.e. URBS) for the destination. | destOutputItems
Pending items | Shows the number of in-flight output transactions and events for the destination. | destPendingItems
Item time at source | Is the commit time of last processed transaction or event for the destination. | destItemTimeAtSource
Messages | Shows the number of output messages sent for the destination. | destMessages
Bytes | Shows the number of bytes in the output messages. | destBytes
Commits | Shows the number of messaging system commits for the destination. | destCommits
Commit time at destination | Is the time of the last commit issued to the messaging system for the destination. | destCommitTimeAtDestination
Pending messages | Shows the number of messages that have been written but not yet committed to the messaging system for the destination. | destPendingMessages
Pending bytes | Shows the number of bytes for the pending messages. | destPendingBytes
De-logged | Shows the number of transactions and events read from the SLOG system file for the destination. | destItemsDeloggedSLOG
Logged | Shows the number of transactions and events written to the SLOG system file for the destination. | destItemsLoggedSLOG
On SLOG | Shows the number of transactions and events currently residing on the SLOG system file for the destination. | destItemsOnSLOG
| destLastMessagingCommitTime minus destLastTransCommitTime | destLastItemDurationTime
| destReplicatedTrans divided by intervalLength | destItemsPerSecond
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength

### Adabas Destination Statistics

#### Sample output

    ADAFI15
      Output items                 :           54,710
       Pending items               :                0
      Item time at source          : 2017-10-17 19:12:36
      BT                           :                0
      ET                           :           21,500
      ET time at destination       : 2017-10-17 19:14:07
      Input: DBID 28857  File    15  Target: DBID 28858  File    15
       A1                          :            6,600
       E1                          :            7,450
       HI                          :                0
       N1                          :                1
       N2                          :            7,450
       Security                    :                1
       S4                          :                1
       UTI                         :                0

#### Labels and values shown for Adabas destination statistics

The Adabas destination statistics section is repeated for each Adabas destination defined to the Event Replicator Server.

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticDestinationAdabasFile | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
ADAFI15 (sample) | Is the Adabas destination name in the above sample. | destinationName
Output items | Shows the number of output transactions and events (i.e. URBS) for the destination. | destOutputItems
Pending items | Shows the number of in-flight output transactions and events for the destination. | destPendingItems
Item time at source | Is the commit time of last processed transaction or event for the destination. | destItemTimeAtSource
BT | Shows the number of BT (backout transaction) commands successfully issued for the destination. | destBTCommands
ET | Shows the number of ET (end transaction) commands successfully issued for the destination. | destETCommands
ET time at destination | Is the time of the last successful ET command that was issued for the destination. | destETTimeAtDestination
| Items de-logged from the SLOG system file. | destItemsDeloggedSLOG
| Items logged to the SLOG system file. | destItemsLoggedSLOG
| Items on the SLOG system file. | destItemsOnSLOG
| destLastETTime minus destLastTransCommitTime | destLastTransactionDurationTime
| destReplicatedTrans divided by intervalLength | destTransactionsPerSecond
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength
Input: DBID nnnnn File xx Target: DBID nnnnn File xx | Is the input database/file and the target database/file combination. | destInputDBID destInputFNR destTargetDBID destTargetFNR
|  | destNumberErrors
A1 | Shows the number of A1 (update) commands issued for the destination. | destUpdateCommands
E1 | Shows the number of E1 (delete) commands issued for the destination. | destDeleteCommands
HI | Shows the number of HI (hold record) commands issued for the destination. | destHoldCommands
N1 | Shows the number of N1 (add) commands issued for the destination. | destInsertCommands
N2 | Shows the number of N2 (add) commands issued for the destination. | destInsertISNCommands
Security | Shows the number of Adabas security operations replicated to the target database/file. | destSecurityFunctions
S4 | Shows the number of S4 (find) commands issued for the destination. | destFindCommands
UTI | Shows the number of Adabas utility operations replicated to the target database/file. | destUtilityFunctions
| Replication method ('ISN ' / 'KEY ') | destReplicationMethod

### Deleted Destination Statistics

#### Sample output

    Deleted destinations
      Output items                 :                1
      Messages                     :                1
       Bytes                       :              110
      Commits                      :                1
      De-logged                    :                1
      Logged                       :                1

#### Labels and values shown for deleted destination statistics

Label | Value Description
--- | ---
Output items | Shows the number of output transactions and events (i.e. URBS) for all deleted destinations.
Messages | Shows the number of output messages sent for all deleted destinations.
Bytes | Shows the number of bytes in the output messages.
Commits | Shows the number of messaging system commits for all deleted destinations.
De-logged | Shows the number of transactions and events read from the SLOG system file for all deleted destinations.
Logged | Shows the number of transactions and events written to the SLOG system file for all deleted destinations.

### Subtask Statistics

#### Sample output

    Output subtask items:
     Main                          :                0
        1                          :              268
        2                          :              101
        3                          :              101

    Output subtask message totals:
     Main                          :                0
        1                          :                1
        2                          :                1
        3                          :                1

The output subtask items detail line is repeated for each output subtask defined to the Event Replicator Server. Each detail line shows the number of output transactions and events (i.e. URBS) processed for the subtask.

The output subtask message totals detail line is repeated for each output subtask defined to the Event Replicator Server. Each detail line shows the number of messages processed for the subtask.

#### Labels and values shown for subtask statistics

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticOutputTask | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
Main (sample) | taskNumber (0 = main task) | taskNumber
items | Shows the number of output transactions and events (i.e. URBS) processed for the subtask. | items
message totals | Shows the number of messages processed for the subtask. | messagesSent
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength

### Input Queue Statistics

#### Sample output

    INPUTQ1
      Input items                  :           54,721
       Pending items               :                0
      Item time at source          : 2017-10-17 19:10:45
      Messages                     :           29,364
       Bytes                       :       26,374,377
      Pending messages             :                1
       Bytes                       :              110
      Commits                      :           29,357
      Commit time at input queue   : 2017-10-17 19:15:45
      Backouts                     :                0

#### Labels and values shown for input queue statistics

The input queue statistics section is repeated for each input queue defined to the Event Replicator Server.

Label | Value Description | Elasticsearch Field
--- | --- | ---
| ReptorStatisticInputQueue | type
| URBS reptor status ("RSTA") | reptorStatus
| Target ID of originating Event Replicator Server | targetID
| Nucleus ID of originating Event Replicator Server | nucleusID
INPUTQ1 (sample) | Is the input queue name in this example. | queueName
Input items | Shows the number of input transactions and events (i.e. URBS) for the input queue. |
Pending items | Shows the number of in-flight input transactions and events (i.e. URBS) for the input queue. |
Item time at source | Is the time of the last input transaction or event processed for the input queue. |
Messages | Shows the number of messages read and committed for the input queue. | queueMessagesReceived
Bytes | Shows the number of bytes in the above messages. | queueBytesReceived
Pending messages | Shows the number of messages read from the input queue but not yet committed. | queuePendingMessages
Bytes | Shows the number of bytes in the above messages. | queuePendingBytes
Commits | Shows the number of commits issued for the input queue. | queueMessagingCommits
Commit time at input queue | Is the time the last commit was issued for the input queue. |
Backouts | Shows the number of back outs issued for the input queue. | queueMessagingBackouts
| Java Date instance containing URBS timeTransProcessedStart equivalent | timestamp
| Java Date instance containing URBS timeTransCommitEnd equivalent | timestampEnd
| interval length in milliseconds timestampEnd minus timestamp | intervalLength

### Deleted Input Queue Statistics

#### Sample output

    Deleted input queues
      Input items                  :                1
      Messages                     :                1
       Bytes                       :              110
      Commits                      :                1
      Backouts                     :                0

#### Labels and values shown for deleted input queue statistics

Label | Value Description
--- | ---
Input items | Shows the number of input transactions and events (i.e. URBS) for all deleted input queues.
Messages | Shows the number of messages read and committed for all deleted input queues.
Bytes | Shows the number of bytes in the above messages.
Commits | Shows the number of commits issued for all deleted input queues.
Backouts | Shows the number of back outs issued for all deleted input queues.

## Event Replicator Target Adapter Statistics

This section describes the various statistics (or metrics) sent from the Event Replicator Target Adapter and made available to the Replication Monitoring subsystem. The Event Replicator Target Adapter has always collected statistics and made them available for display using Event Replicator Target Adapter Administration. These statistics are described in the Event Replicator Target Adapter documentation at Reviewing Event Replicator Target Adapter Statistics.

The statistics collected and displayed by the Event Replicator Target Adapter and its Administration tool are now also sent to Elasticsearch and made available to Kibana dashboards and visualizations as part of Replication Monitoring. The "Category Type" in the Event Replicator Target Adapter statistics documentation maps directly to the Elasticsearch fields "type" and "id". Some examples of this mapping are shown in the table below:

Category Type | Elasticsearch “type” value | Elasticsearch “id” value
--- | --- | ---
/engine/… | engine | engine id e.g. "Engine-0"
/source/sourcename | source | sourcename e.g. "HOST3-S1"
/jdbc/targetname | jdbc | targetname e.g. "host05 SQL Server"
/subscription/subname | subscription | subname e.g. "EAPSUB3"
/table/tablename | table | tablename e.g. "Employees"
/heartbeat/destname | heartbeat | destname e.g. "DEST2"
/heartbeat_etb/destname | heartbeat_etb | destname e.g. "DEST2"
/heartbeat_processing/destname | heartbeat_processing | destname e.g. "DEST2"
/heartbeat_slog/destname | heartbeat_slog | destname e.g. "DEST2"
/latency/source.target.subname | latency | source.target.subname e.g. "HOST3-S1.host05 SQL Server.EAPSUB3"

Please refer to Reviewing Event Replicator Target Adapter Statistics for further details.

## Heartbeat

The statistics reported by the Event Replicator Target Adapter include a Heartbeat functionality that merits some additional description. A heartbeat is a special short message generated and sent down the pipe through the messaging system and into the Target Adapter. The purpose of this message is to ensure that all pieces of the replication system are working properly at a regular interval. The Target Adapter recognizes these heartbeat messages and reports statistics about the flow timing as part of Replication Monitoring.

The full set of statistics that includes all properties of the heartbeat is written at the conclusion of all heartbeat processing as an Elasticsearch data point with field "type" set to "heartbeat". Additional data points are written containing certain sub-types that comprise the full heartbeat. These are written with field "type" set to "heartbeat_etb", "heartbeat_processing", and "heartbeat_slog". All data points contain a "timestamp" field with different values appropriate for the sub-type. The final full heartbeat data point is written with "timestamp" set to the final time when all heartbeat statistics have been fully recognized and processed. Some important points in the heartbeat timeline include:

Timestamp Name | Timestamp Description
--- | ---
creationTime | Time the URBS heartbeat message was created on the mainframe by Event Replicator for Adabas.
readFromSlog | Time the URBS heartbeat message was read from the SLOG file on the mainframe by Event Replicator for Adabas. This timestamp will only have a value if the mainframe processing load was sufficiently high to cause the message to be written to the SLOG file for later processing.
infoTime | Time when Unit-of-Work status was obtained from EntireX Broker by Event Replicator Target Adapter.
artTime | Time when the heartbeat message is fully received and recognized by Event Replicator Target Adapter.

The table below shows all heartbeat "type" values with the "timestamp" value used and the list of Elasticsearch fields included in the data points for that sub-type.

Elasticsearch “type” value | Elasticsearch “timestamp” value | Elasticsearch fields
--- | --- | ---
heartbeat_etb | infoTime | maxBytes maxMessages newestItem numberBytes numberMessages numberUows oldestItem
heartbeat_processing | infoTime | processingTime
heartbeat_slog | readFromSlog | slogTime
heartbeat | artTime | Complete field list described at Reviewing Event Replicator Target Adapter Statistics in table entry Heartbeat.

Please consult Reviewing Event Replicator Target Adapter Statistics for more details.

### On this page

- Event Replicator Server Statistics
- Event Replicator Target Adapter Statistics
- Heartbeat