Troubleshooting Tips: Monitoring Performance
I see performance degradation when the Log Invocation policy is configured to log API Data Store or an external Elasticsearch
When the Log Invocation policy is configured for APIs, API Gateway displays only some transaction events in the Analytics tab of an API, while ignoring others. This issue occurs during times of high transaction event loads.
Resolution:
Increase the following extended settings and watt properties values to get the desired performance improvements:
events.collectionPool.maxThreads
events.collectionPool.minThreads
events.collectionQueue.size
events.reportingPool.maxThreads
events.reportingPool.minThreads
events.reportingQueue.size
Increase the following JVM heap size in the wrapper.conf file located atSAGInstallDir\profiles\IS_default\configuration:
wrapper.java.initmemory
wrapper.java.maxmemory
I see that the runtime events widget does not render transaction events with a huge payload
Runtime events widget under API-specific analytics dashboard does not display the transactional events and a shard failure error appears when there are huge Base64 encoded payloads.
Resolution:
To avoid encountering errors while handling huge Base64 encoded payloads:
Set the
index.highlight.max_analyzed_offset index level setting to maximum.
Sample to set the index.highlight.max_analyzed_offset index level setting to maximum is as follows:
curl -X PUT "localhost:9240/<index_name>/settings?pretty" -H 'Content-Type: application/json' -d' { "index.highlight.max_analyzed_offset" :50000000
}
Set the
indices.query.bool.max_clause_count property value to 2048 in the elasticsearch.yml file located at
SAG_Installdir/InternalDataStore/config.
Increasing the indices.query.bool.max_clause_count property value to 2048 does not impact the transactional flow performance. But by increasing this value, you are allowing API Data Store to run queries and aggregations over many fields which might consume more resources to run these queries.
I see that I am not able to substitute credentials when using Kibana keystore
Substituting credentials when using Kibana keystore does not work in the API Gateway UI. When navigating to the API Gateway Analytics page, the user is asked for Elasticsearch credentials in an authentication window.
Resolution:
To resolve this issue, use plain text passwords while authenticating Kibana keystore.
I see the low disk space issue, and API Gateway stops working for the WRITE operations.
This error occurs when there is low disk space.
The following error messages are seen in the SAG_EventDataStore.log file in the SAGInstallDir\InternalDataStore\logs :
Exception: [WARN ] [o.e.c.r.a.DiskThresholdMonitor] [localhost1568897216386] flood stage disk watermark [95%] exceeded on [BOf6SQe2SwyI93vi4RlBNQ] [localhost1568897216386] [C:\SoftwareAG\InternalDataStore\data\nodes\0] free: 2.4gb [2.4%], all indices on this node will be marked read-only.
Saving an API -> error message ("Saving API failed. com.softwareag.apigateway.core.exceptions.DataStoreException: Error while saving the document. doc Id - 6d5c7ac0-574a-4a53-acba-a738f21e3142, type name - _doc, message - "index [gateway_default_policy] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];" ")
Resolution:
Increase the disk space in all nodes or clean up disk space by clearing unwanted data. If you do not perform this step, then there is a chance that this error might appear again.
Make a REST call to:
curl -XPUT -H "Content-Type: application/json"
http://localhost:9200/_all/_settings -d '
{"index.blocks.read_only_allow_delete": null}
You can optimize the usage of disk space using the Watermark property of Elasticsearch. For information about the property, see https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html