Upgrading Elasticsearch in API Gateway
When considering an upgrade to Elasticsearch, it is necessary to ensure compatibility between the API Gateway and both the current and target versions of Elasticsearch. This section explains how to perform Elasticsearch upgrades to ensure both backward and forward compatibility.
Note:
Software AG recommends maintaining two separate Elasticsearch data stores – one for core data and the other for transactional data for better performace of the application.
Update scenarios
To maintain the security posture of the API Gateway, it is recommended to perform periodic updates of Elasticsearch throughout the lifecycle of the API Gateway version. Regular updates help address newly identified security vulnerabilities and enhance the overall reliability of the system.
Note:
1. Instead of undergoing a data copy process, you must restore data from an existing Elasticsearch backup. Following the restoration of Elasticsearch data, ensure connectivity between the External Elasticsearch version 8.12.2 and Kibana 8.12.2 to API Gateway.
2. After upgrading to Elasticsearch 8.12.2, back up Elasticsearch data to an alternative storage location regularly. In case of data loss or damage, use the backup copy for recovery.
Overview of data copy process
The data copy process involves transferring Elasticsearch data securely from a source to a destination endpoint. This overview outlines the key steps required for a successful and secure data copy operation:
1. Copy the Elasticsearch data.
2. Perform data copy for secured Elasticsearch.
3. Check the status of the data copy process.
4. Verify the configurations.
Step 1. Copy the Elasticsearch data
When upgrading from one Elasticsearch version to the next, it is necessary to transfer the core and transcational data from the previous version to the target version. Perform the following steps to copy Elasticsearch data to the target destination:
1. Open your REST client.
2. Provide the details of the Elasticsearch endpoints for both the core data and the analytics data in the Body section.
Configure the following parameters in the REST client by providing the required values. These parameters define the target endpoint where the Elasticsearch data should be copied.
targetCoreESEndpointConfig: Specifies the configuration for the target core Elasticsearch endpoint. By specifying the values for this parameter, only the core Elasticsearch data is copied to the target endpoint. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the target core Elasticsearch endpoint. | "endpoint":"http://targetendpoint:9200" |
username | Specifies the username for authentication. | "userName":"abc" |
password | Specifies the password for authentication. | "password":"xyz" |
targetEventsAndLogsESEndpointConfig: Specifies the configuration for the target events and logs Elasticsearch endpoint. By specifying the values for this parameter, only the transactional Elasticsearch data is copied to the target endpoint. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the target events and logs Elasticsearch endpoint. | "endpoint":"http://targetendpoint:9200" |
username | Specifies the username for authentication. | "userName":"xyz" |
password | Specifies the password for authentication. | "password":"abc" |
targetDashboardConfig: Specifies the configuration for the target Dashboard endpoint Elasticsearch endpoint. By specifying the values for this parameter, the endpoint is directed to the target Kibana dashboard, which has the analytics data.To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the target Dashboard endpoint. | "endpoint":"http://targetendpoint:5601" |
{
"targetCoreESEndpointConfig": {
"endpoint": "http://targetendpoint:9200",
"userName": "xyz",
"password": "abz",
},
"targetEventsAndLogsESEndpointConfig": {
"endpoint": "http://targetenpoint:9200",
"userName": "xyz",
"password": "abc",
},
"targetDashboardConfig": {
"endpoint":"http://targetendpoint:5601"
}
}
Copy the core data individually by configuring and providing the values for the
'targetCoreESEndpointConfig' parameter in the REST client and making a POST call to the follwing URL:
POST/rest/apigateway/migrate/datacopy/core
Similarly, copy the events and logs data individually by configuring and providing the values for the 'targetEventsAndLogsESEndpointConfig' and 'targetDashboardConfig' parameters in the REST client and making a POST call to following URL:
POST/rest/apigateway/migrate/datacopy/eventsandlogs
Step 2. Perform data copy for secured Elasticsearch
Note:
This step is only applicable if you are using a secured Elasticsearch. Proceed with Step 3 if you are not using a secured Elasticsearch.
1. To communicate with a secured Elasticsearch using Keystore and Truststore, configure the following parameters in the REST client by providing the required values:
keystoreConfig: Specifies the configuration for the target Keystore endpoint. By specifying the values for this parameter, the target Elasticsearch server validates the client certificate and key stored in the
Keystore. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
filepath | Specifies the file path of the target core/events and logs Keystore endpoint. | "filePath": "c:/keystorefiles/coreelasticks.jks" or "filePath": "c:/keystorefiles/eventelastic.jks" |
alias | Specifies the alias for authentication. | "alias":"abc" |
password | Specifies the password for authentication. | "password":"xyz" |
truststoreConfig: Specifies the configuration for the target Truststore endpoint. The
Truststore validates the target Elasticsearch server certificate. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
filepath | Specifies the file path of the target core/events and logs Truststore endpoint. | "filePath": "c:/truststorefiles/coreelasticts.jks" or "filePath": "c:/truststorefiles/eventelasticts.jks" |
password | Specifies the password for authentication. | "password":"xyz" |
For example,
{
"targetCoreESEndpointConfig": {
"endpoint": "https://targetendpoint:9200",
"userName": "xyz",
"password": "abz",
"keystoreConfig": {
"filePath": "c:/keystorefiles/coreelasticks.jks",
"alias": "hello",
"password": "abcd"
},
"truststoreConfig": {
"filePath": "c:/truststorefiles/coreelasticts.jks",
"password": "abcd"
}
},
"targetEventsAndLogsESEndpointConfig": {
"endpoint": "https://targetendpoint:9200",
"userName": "xyz",
"password": "abc",
"keystoreConfig": {
"filePath": "c:/keystorefiles/eventelastic.jks",
"alias": "helloa",
"password": "abcc"
},
"truststoreConfig": {
"filePath": "c:/truststorefiles/eventelasticts.jks",
"password": "abcc"
}
},
"targetDashboardConfig": {
"endpoint":"https://targetendpoint:5601"
}
}
2. Make a call to the following URL, with the IP address of the system where API Gateway is installed and press Enter:
POST/rest/apigateway/migrate/datacopy
The data copy process will start.
Step 3. Check the status of the data copy process
Throughout the upgrade process, you can retrieve the status, which can be valuable for troubleshooting any potential issues.
You can use one of the following ways to get the status of the Elasticsearch data copy.
1. Using GET call to get the status
a. Open your REST client.
b. Provide the details of the Elasticsearch endpoints for both the core data and the analytics data.
c. Make a call to the following URL, with the IP address of the system where API Gateway is installed and press Enter:
GET/rest/apigateway/migrate/datacopy/status
Invoke the URL at regular intevals to get the latest status.
2. Using webhook to get the status
You can use webhook to automate the status delivery using an application. For example, you can use Microsoft Teams to get the timely status of the data copy process whenever API Gateway undergoes migration. Perform the following tasks to use webhook to get the updates:
a. Open your REST client.
b. Provide the details of the Elasticsearch endpoints for both the core data and the analytics data in the Body section.
In addition to the 'targetCoreESEndpointConfig', 'targetEventsAndLogsESEndpointConfig' and 'targetDashboardConfig', you can configure the following parameter in the REST client by providing the required values.
webhookConfig: Specifies the configuration for the webhook notification. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the webhook endpoint. | "https://abc.webhook.office.com/webhookb2/abcd |
username | Specifies the username for authentication (if required). | "userName":"xyz" |
password | Specifies the password for authentication (if required). | "password":"abc" |
template | Specifies the JSON template for the message card. | "{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"@@status@@\"}, \"value\":\"@@workflow@@\"}\ |
Provide three placeholders - @@status@@, @@workflow@@ and @@log@@.
Get the status of the core data copy process individually by configuring and providing the values for the targetCoreESEndpointConfig and webhookConfig parameters in the REST client and making a POST call to the follwing URL:
POST/rest/apigateway/migrate/datacopy/core
Similarly, get the status of the events and logs data copy process individually by configuring and providing the values for the targetEventsAndLogsESEndpointConfig, targetDashboardConfig and webhookConfig parameters in the REST client and making a POST call to the following URL:
POST/rest/apigateway/migrate/datacopy/eventsandlogs
c. Make a call to the following URL, with the IP address of the system where API Gateway is installed and press Enter:
POST/rest/apigateway/migrate/datacopy/status
Note:
You can use the GET call to get updates even if you use webhook to get regular updates.
Step 4. Verify the configurations
Upon successful data copying, the configuration files is automatically reflect the changes by pointing to the new cluster. You can verify the same in the following files:
SAGInstaller/Integration Server/default/packages/WmAPIGateway/config/resources/elasticsearch/config.properties SAGInstaller/Integration Server/default/packages/WmAPIGateway/config/resources/elasticsearch/analyticsds.properties Rolling back the the source endpoints
In the event of an unsuccessful upgrade, a rollback option is available. The configuration changes is reverted, and the client is directed back to the source Elasticsearch cluster. Perform the following steps to initiate the rollback process:
1. Open your REST client.
2. Provide information regarding source end points for each Elasticsearch in the Request section.
Configure the following parameters in the REST client by providing the required values. These parameters define the target endpoint where the Elasticsearch data should be copied.
targetCoreESEndpointConfig: Specifies the configuration for the source core Elasticsearch endpoint. By specifying the values for this parameter, only the core Elasticsearch data is copied to the source endpoint. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the source core Elasticsearch endpoint. | "endpoint":"http://sourceendpoint:9200" |
username | Specifies the username for authentication. | "userName":"abc" |
password | Specifies the password for authentication. | "password":"xyz" |
targetEventsAndLogsESEndpointConfig: Specifies the configuration for the source events and logs Elasticsearch endpoint. By specifying the values for this parameter, only the transactional Elasticsearch data is copied to the source endpoint. To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the source events and logs Elasticsearch endpoint. | "endpoint":"http://sourceendpoint:9200" |
username | Specifies the username for authentication. | "userName":"xyz" |
password | Specifies the password for authentication. | "password":"abc" |
targetDashboardConfig: Specifies the configuration for the source Dashboard endpoint Elasticsearch endpoint. the endpoint is directed to the source Kibana dashboard, which has the analytics data.To configure this parameter, you can use the following settings by providing the required values.
Settings | Description | Example |
endpoint | Specifies the URL of the source Dashboard endpoint. | "endpoint":"http://sourceendpoint:5601" |
For example,
{
"targetCoreESEndpointConfig": {
"endpoint": "http://sourceendpoint:9200",
"userName": "xyz",
"password": "abz",
},
"targetEventsAndLogsESEndpointConfig": {
"endpoint": "http://sourceenpoint:9200",
"userName": "xyz",
"password": "abc",
},
"targetDashboardConfig": {
"endpoint":"http://sourceendpoint:5601"
}
}
You can provide the source Keystore/Truststore values to get the source data.
3. Make a call to the following URL, with the IP address of the system where API Gateway is installed and press Enter:
POST/rest/apigateway/migrate/datacopy/rollback