The following error message appears when API Gateway fails to start in a cluster setup:

Elasticsearch is not ready to serve the requests.

The error message appears only when the API Data Store cluster health status is not green or yellow. This means that one of the API Data Store node is not started properly or errors occurred in cluster. The package initialization might fail due to this.

Resolution:

Check the cluster health status of API Gateway using the following REST endpoint GET /rest/apigateway/health/engine to identify the problem. If the cluster health status is red, it means that errors occurred in cluster or one of the API Data Store node is not started properly. Errors might occur in API Data Store when there is a problem with its configurations. For more information about the cluster health status and its configurations, see Monitoring API Gateway , and API Gateway Cluster Configuration.

On restarting API Gateway nodes, the configured Elasticsearch cluster settings for disk-based shard allocation properties are reset.

Resolution:

To avoid reset of the shard allocation properties across restarts, use persistent keyword in the Elasticsearch cluster settings.

Sample for persistent keyword usage in Elasticsearch cluster settings is as follows:

Elasticsearch does not start in a clustered environment with the MasterNotDiscoveredException exception.

Resolution:

To resolve this issue include the cluster.initial_master_nodes property in the elasticsearch.yml file located at SAG_Installdir /InternalDataStore/config.

Sample for cluster.initial_master_nodes property in elasticsearch.yml is as follows:

Obfuscating Elasticsearch credentials in elasticsearch.yml does not work for API Gateway versions 10.5 and above.

Resolution:

After the Elasticsearch 7.0 upgrade, there is a limitation to obfuscate Elasticsearch credentials in API Gateway.

As a workaround you can use the Environment variable substitution method that Elasticsearch provides to obfuscate Elasticsearch credentials.

For more information, see Elasticsearch documentation.

One of the reasons for the dangling index problem is that there are two or more API Gateway instances, which are used independently (standalone) for a while and then they are brought into a cluster. When this happens, both the API Data Store nodes try to update each other on the same indexes. As both the API Data Store nodes have the same indexes with different data, this leads to the dangling index error.

The following error message appears in any of the indexes:

[ WARN ] [o.e.g.DanglingIndicesState] [s1gp-igw01-gsb.sgi-idm.fednet.intra1566308686237] [[gateway_default_cache/rSspQXrrRKGdQCwS8KaYHg] ] can not be imported as a dangling index, as index with same name already exists in cluster metadata.

When this issue occurs, you can notice inconsistent data in the API Data Store nodes after every restart.

Resolution:

To resolve an existing dangling cluster, contact Software AG Global Support.