API Gateway 10.11 | Upgrading and Migrating to API Gateway | Upgrading API Gateway | Troubleshooting Tips
 
Troubleshooting Tips
I see that a scheduled API Gateway upgrade fails due to the re-indexing issue.
Upgrading API Gateway fails due to the re-indexing issue in the migration process.
Sample error is as follows:
Data store migration failed.
Reason - Error while reindexing APIs. Error type - illegal_argument_exception, reason Remote responded with a chunk that was too large. Use a smaller batch size.
Resolution:
To resolve this issue, modify the value of the apigateway.migration.batchSize property in the migration.properties file located at SAG_Installdir\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate.
The default value of the apigateway.migration.batchSize property is 100.
Modify the value of the apigateway.migration.batchSize property based on the size of the payload. If you have a huge payload, reduce the apigateway.migration.batchSize property value to a smaller number. For example, apigateway.migration.batchSize=25.
The migration time increases, if you reduce the apigateway.migration.batchSize property value.
I see that the data store migration fails when I provide hostname instead of localhost in the elasticsearch.yml file.
My data store migration fails when I provide hostname instead of localhost in elasticsearch.yml.
Basically, I am adding the following line in the file <installDir>\InternalDataStore\config\elasticsearch.yml before migration.
reindex.remote.whitelist:hostName:9240
It works fine with localhost, but with actual hostName it fails with the following error,
Backup process started for the configured elasticsearch host [localhost:8240] Creating repository [migration_default] 2019-06-13 17:00:41 ERROR AbstractElasticsearchClient:98 - APIGW Migration Exception:Error while creating repository - migration_default. Message - listener timeout after waiting for [100000] ms java.io.IOException: listener timeout after waiting for [100000] ms at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:660) ~ [elasticsearch-rest-client-5.6.4.jar:5.6.4]
Resolution:
You can use regular expressions instead of using the exact hostname. To learn more about how to set regex values for the reindex.remote.whitelist in the elasticsearch.yml file, see Elasticsearch documentation.
I see an Elasticsearch exception while migrating an API Gateway instance using direct standalone mode.
Upgrading API Gateway using direct standalone mode fails with an Elasticsearch exception.
The Elasticsearch exception is as follows:
ElasticsearchException[Error while reindexing APIs. Error type - illegal_argument_exception,reason [localhost:1240] not whitelisted in reindex.remote.whitelist]
Reason: Remote re-indexing property is missing in the target elasticsearch.yml file.
Resolution:
Add the reindex.remote.whitelist property in the target elasticsearch.yml file located at <TARGET>\InternalDataStore\config.
Syntax for reindex.remote.whitelist property in the target elasticsearch.yml file is as follows:
reindex.remote.whitelist: <source_host>:<source_port>
I see a No such command exception while migrating an API Gateway instance using direct standalone mode.
Upgrading API Gateway using direct standalone mode fails with a No such command exception.
The exception is as follows:
Exception thrown during migration operation. Exiting the operation. No Such command - -srcDir
Resolution:
The API Data Store and API Gateway argument must be passed in the following migration utility command:
migrate.bat apigateway -srcDir <SOURCE> -instanceName <source instance name> -newInstanceName <target instance name>
Sample migration utility command is as follows:
migration.bat apigateway -srcDir C:\SoftwareAG_10.1 -instanceName default -newInstanceName default
I see java.io.FileNotFoundException exception while migrating an API Gateway instance using backup standalone mode.
Upgrading API Gateway using backup standalone mode fails with a java.io.FileNotFoundException exception.
The exception is as follows:
Deleting the backup folder before migration... Migration Process 0% [>] 0/100 (0:00:00) java.io.FileNotFoundException: 15:41:13.580 [main] ERROR com.softwareag.apigateway.utility.command.backup.instance.BackupApiGatewayInstance - Error occurred while trying to parse the Manifest file to obtain the version information. java.io.FileNotFoundException:(The system cannot find the file specified)
Reason: Existence of isExtract file due to migration failure.
Resolution:
Delete the isExtract folder in the directory C:\<SAGInstallDir>\.
I see an Elasticsearchclient bean creation exception while migrating an API Gateway instance.
Upgrading API Gateway fails with an Elasticsearchclient bean creation exception.
Resolution:
Make sure that all the respective Elasticsearch nodes are up and running.
I see a RepositoryVerificationException while migrating API Gateway cluster using backup mode.
Upgrading API Gateway fails with a RepositoryVerificationException exception.
The exception is as follows:
[2018-12-19T12:34:03,892][WARN ][o.e.r.VerifyNodeRepositoryAction][SAG-2KXGBH2.eur.ad.sag1544697848038] [default] failed to verify repositoryorg.elasticsearch.repositories.RepositoryVerificationException: [default] a file written by master to the store [C:\SoftwareAG_10.3\InternalDataStore\dasoSnap\default] cannot be accessed on the node [{SAG-2KXGBH2.eur.ad.sag1544697848038} {yL84xqhZQSuFfUmj0GrFbQ}{uGv-BmTBT6e8LCpOsxpGOg} {10.60.37.18}{10.60.37.18:9340}]. This might indicate that the store [C:\SoftwareAG_10.3\InternalDataStore\dasoSnap\default] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node
Resolution:
Make sure that the path.repo property is a single common location reachable and accessible by all other nodes.
Configure a single common shared path for the path.repo property in the elasticsearch.yml file located at <SAGInstallDir>\InternalDataStore\config.
I see a system cannot find the path specified error while migrating API Gateway.
Upgrading API Gateway fails with a system cannot find the path specified error.
Resolution:
Make sure that you are running the .bat file from a proper location.
I see a content too long exception while upgrading API Gateway.
Upgrading API Gateway fails with a content too long exception.
The exception is as follows:
2019-10-16 11:59:36 ERROR ESDataStoreHandler:327 - {"type":"illegal_argument_exception","reason":"Remote responded with a chunk that was too large. Use a smaller batch size.","caused_by":{"type":"content_too_long_exception","reason":"entity content is too long [185463385] for the configured buffer limit [104857600]"}}
Reason: The size of the documents selected for re-indexing as per the batch size configuration is large.
Resolution:
Try with a smaller document batch size number.
I see an error while getting task details when upgrading API Gateway.
Upgrading API Gateway fails with an error while getting task details.
Sample error is as follows:
Error while getting task details for task id - 6L6RMNEOQF64lQbNeI7J_g:683354. Message - task [6L6RMNEOQF64lQbNeI7J_g:683354] isn't running and hasn't stored its results
Reason: When the disk space in the machine is less than 10%, Elasticsearch marks the index as read-only and the re-index task fails abruptly.
Resolution:
Increase the memory size to resolve this issue.
I see an error while restoring the index when upgrading API Gateway.
Upgrading API Gateway fails with an error while restoring the index, when migrating a standalone API Gateway instance or an API Gateway cluster using backup mode.
Sample error is as follows:
ERROR ESDataStoreHandler:432 - Error while restoring index - gateway_default_analytics java.io.IOException: listener timeout after waiting for [100000] ms
Reason: While restoring the Elasticsearch data from backup snapshot, the elasticsearch client is timed out due to the time taken to restore the large data in Elasticsearch.
Resolution:
To resolve this issue, increase the value of the socket timeout and maximum retry timeout properties accordingly.
socketTimeout
Read timeout in milli seconds between API Gateway and API Data Store.
Property: pg.gateway.elasticsearch.http.socketTimeout
maxRetryTimeout
Time out in milli seconds to wait for retries.
Property: pg.gateway.elasticsearch.http.maxRetryTimeout
It is advisable to set the maximum retry time for a request to (number of nodes * socketTimeOut )+ connectionTimeout.