Troubleshooting Tips: API Gateway Destination

I see that the API Gateway transactions are not logged in API Data Store when there is a medium to high load that is 70 to 400 Transactions per second (TPS)

If you have configured an external API Date Store destination, ensure that the configuration is right. This problem might occur when the API Data Store configurations are not set properly to handle the load.

Set the following configurations to the properties in config.properties file located at SAG_Install_Directory\IntegrationServer\instances\IS_Instance_Name\packages\WmAPIGateway\config\resources\elasticsearch\config.properties to meet the following throughput values: Up to 2000 TPS, Up to 500 GB storage utilization, and less than <500 ms native service latency:

pg.gateway.elasticsearch.hosts = Elasticsearch Service endpoint, that is localhost:9240
pg.gateway.elasticsearch.http.connectionTimeout = 10000
pg.gateway.elasticsearch.http.socketTimeout = 30000
pg.gateway.elasticsearch.sniff.enable = false
pg.gateway.elasticsearch.http.maxRetryTimeout = 100000

You can verify the transactional events count using http://<hostname>:<es_port>/testindex/transactionalEvents/count

Note:
You can modify the configurations according to your business requirements.

For more information, see Hardware and Product Configurations.

I see that I am not able to parameterize custom indices for transaction events in API Gateway

Custom indices can be configured for transaction events in API Gateway under Destinations > Elasticsearch > Configuration page. But parameterizing custom indices is not supported in API Gateway. For example, appending the current date to the index name. Consider an index name txnData and it must be appended with the current date, that is txnData_12_08_2020 and rolled up daily.

You can achieve this use case using the Elasticsearch aliases and rollover APIs of Elasticsearch.

A sample use case is as follows:

Create an alias in the Elasticsearch server, for example txnData and an index rollover like txnData_12_08_2020, txnData_12_08_2020. The alias txnData should point to all the indices txnData_12_08_2020, txnData_12_08_2020. One of the indices in the list is write-enabled and that are the latest and rest of the indices is read-only.

API Gateway sends the events to txnData alias and that alias defines the index that it should write the data to, based on the date.