Software AG Products 10.11 | Administrating API Gateway | Operating API Gateway | Monitoring API Gateway | Node-level Monitoring | Application Monitoring | Monitoring API Data Store | Application Metrics
 
Application Metrics
Monitor the following metrics to analyze API Data Store health.
*Index Size
*Cluster Health
*Number of Shards
*GC Monitoring
Note:
The threshold values, configurations, and severities that are mentioned throughout this section are the guidelines that Software AG suggests for an optimal performance of API Data Store. You can modify these thresholds or define actions based on your operational requirements.
For details about how to generate thread dump and heap dump, see Troubleshooting: Monitoring API Data Store.
If the metrics return an exceeded threshold value, consider the severity as mentioned and perform the possible actions that Software AG recommends to identify and debug the problem and contact Software AG for further support.
Index Size
When data on a particular index exceeds a certain limit, it is essential to rollover and create a new index. In API Gateway, it is essential to monitor the indices for transactional events. For transactional events, you must rollover the index, when the index size exceeds 25 GB. When an index is rolled over, a new index is created with two primary and a replica for each shard. The naming convention of the new index is gateway_default_analytics_transactionalEvents_YYYYMMDDHHMM.
For information on creating a rollover, see Creating Rollover of an Index.
API Data Store Cluster Health
To ensure optimal health and performance of API Data Store, Software AG recommends monitoring the API Data Store cluster health regularly.
Command
Description
curl -X GET http://localhost:9240/_cluster​/health?pretty
This command retrieves API Data Store cluster health status.
$.status
This JSON path expression retrieves the cluster health status from the response.
$.number_of_nodes
This JSON path expression retrieves the number of nodes in the cluster from the response.
The response JSON of the health check request displays a status field in the response. The status can have the values green, yellow or red. The cluster health status is displayed based on the following color codes:
Status
Description
green
Indicates that the cluster is in a healthy state. When API Data Store is handling huge data, it takes some time to display the cluster health status.
yellow
Indicates that the cluster is not in a healthy state. Identify the cause and rectify it. During this time, API Data Store processes the requests for the index that is available. If there are unassigned shards, then identify the unassigned shards, check the reason for the unallocation and resolve the issue.
*Run the following command to retrieve the list of unassigned shards.
curl -X GET “http://localhost:9240/_cat/
shards?h=index,shard,primaryOrReplica,state,docs,store,ip,​node,segments.count,unassigned​.at,unassigned.details,unassigned.for,​unassigned.reason,help,s=index&v”
*Run the following command to check the unallocated reason for specific shards.
curl -X GET "http://localhost:9240/_cluster/allocation/
explain" -d ‘{ "index" :"index name","primary" : "true|false","shard": "shardnumber"}’reason,help,s=index&v”
red
Indicates that API Data Store nodes are down or not reachable or the API Data Store master is not discovered. If the number of nodes does not match the number of API Data Store nodes configured, identify the node that did not join the cluster and identify the root cause for the node to not join the cluster. Based on the root cause, identify if your API Data Store is down. If your API Data Store is down and not reachable, check the connectivity.
A sample HTTP response is as follows:

{
"cluster_name": "SAG_apidatastore_cluster",
"status": "green",
"timed_out": false,
"number_of_nodes": 3,
"number_of_data_nodes": 3,
"active_primary_shards": 101,
"active_shards": 202,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100.0
}
The overall cluster status is green since all API Data Store nodes work as expected.
Number of shards
To ensure proper allocation of shards to nodes, Software AG recommends to monitor the number of shards regularly.
Command
Description
curl -X GET "http://localhost:9240/_cluster​/health?pretty"
This command retrieves the number of shards on API Data Store.
If the total number of active shards from the response exceeds the heap space * nodes * 20 count, then increase the heap space of API Data Store nodes or add a new API Data Store node. For more information on adding a new API Data Store node, see Adding New Nodes to an Elasticsearch Cluster.
API Data Store considers a maximum of 20 active shards per GB of heap space as healthy. Perform any of the following actions to maintain the total number of active shards:
*Scale up the API Data Store node.
*If you are not able to scale up the API Data Store node, then increase the heap size as the last option. The heap space should not be more than half of system memory (RAM). For example, if the system memory is 16 GB, you can allocate a maximum of 8 GB for API Data Store.
To increase the heap space, modify the parameters Xms2g and Xmx2g in the jvm.options file located at SAG_Install_Directory\InternalDataStore\config.
Garbage Collection (GC) Monitoring
The GC metric provides the GC run-time in seconds. You must check GC run-time once every five minutes. The average GC run-time should not exceed one second.
Metric
Description
*elasticsearch_jvm_gc_​collection_seconds_sum
*elasticsearch_jvm_gc_​collection_seconds_count
The quotient of both the metrics gives the GC run time.
If the quotient is more than 1 second, it implies that GC is taking longer time to run and this slows down API Data Store request processing. You must collect the logs and get the mapping of API index and transaction index.