Using the Externalized Configuration Files

Software AG Products 10.5 | Configuring API Gateway | API Gateway Configuration | Externalizing Configurations | Using the Externalized Configuration Files

The API Gateway administrator provides configuration settings in one or more external configuration files and creates the master configuration file listing the external configuration files. On startup, API Gateway reads config-sources.yml file and loads all the external configuration source files that it references. The settings in the externalized configuration files override the respective internal configuration settings (such as uiconfiguration.properties, server.cnf). Once the API Gateway configuration space is updated, the rest of the API Gateway package gets loaded with the updated configuration settings.

Note:
For settings that are not given in the externalized configuration files, API Gateway use the settings given in the internal configuration files.

The below sample externalized configuration file template contains the configuration settings that the API Gateway administrator wants to externalize. The given external configuration settings overwrite the respective internal configuration settings. For the configuration settings that are not specified in the externalized configuration file, the settings given in the respective internal configuration files take precedence.

apigw:
elasticsearch:
.....
kibana:
.....
filebeat:
......
cluster:
......

Elasticsearch Configuration

Note:
Install and run Elasticsearch, version 7.2, if you are configuring an external instance of Elasticsearch.

The Elasticsearch configuration and details section contains all the necessary properties for an Elasticsearch HTTP client using which API Gateway connects to either an externally running Elasticsearch server or to the Elasticsearch-powered API Gateway data store in API Gateway. The key configurations are as follows:

tenantId. The API Gateway tenant id, using which the Elasticsearch indices are created for that tenant.

hosts. A comma separated list of external Elasticsearch instances. Example: host1:9200,host2:9240.

autostart. Optional. If the value is not provided, by default it would be false. API Gateway would connect to the given external Elasticsearch hosts. If the value is set to true, API Gateway automatically starts the API Gateway data store. In this case, the hosts should point to API Gateway data store host and port. The default host for the API Gateway data store is localhost:9240.

http. The basic authentication credentials and HTTP-connection specific properties.

https. If the enabled property within https is set to true, API Gateway uses the other https properties to connect to the configured hosts.

sniff. These properties help in adding a new Elasticsearch node to the Elasticsearch cluster.

outboundproxy. Outbound proxy settings between API Gateway and Elasticsearch.

clientHttpResponseSize. Maximum Response payload size in MB.

A sample Elasticsearch configuration is as follows:

apigw:
elasticsearch:
tenantId: apigateway
hosts: localhost:9200
autostart: false
http:
username: elastic
password: changeme
keepAlive: true
keepAliveMaxConnections: 10
keepAliveMaxConnectionsPerRoute: 100
connectionTimeout: 1000
socketTimeout: 10000
maxRetryTimeout: 100000
https:
enabled: false
keystoreFilepath: C:/softwares/elasticsearch/config/keystore-new.jks
truststoreFilepath: C:/softwares/elasticsearch/config/truststore-new.ks
keystoreAlias: root-ca
keystorePassword: 6572b9b06156a0ff778c
truststorePassword: manage
enforceHostnameVerification: false
sniff:
enable: false
timeInterval: 1000
outboundProxy:
enabled: false
alias: somealias
clientHttpResponseSize: 1001231

Kibana Configuration

Note:
Install Kibana, version 7.2, if configuring an external instance of Kibana.

The Kibana configuration supports setting the Kibana server URL, which can point to either the one that is run by API Gateway or any externally running server. It also contains the SSL certificate related settings that would be used to connect to the SSL protected Elasticsearch server. The key configurations are as follows.

dashboardInstance. The Kibana server URL in the format scheme://hostname:port. Example: http://vmabc:5601.

autostart. Optional. If the value is not provided, by default it would be false. API Gateway would connect to the given external Kibana server. If the value is set to true, API Gateway automatically starts the internal Kibana server. In this case, the hosts should point to internal Kibana server host and port. The default value is http://localhost:9405.

sslCA. A list of paths to the PEM file for the certificate authority for the Elasticsearch instance.

sslCert. The path to the PEM format certificate for SSL client authentication.

sslKey. The client certificate key used for client authentication. These files are used to verify the identity of Kibana to the Elasticsearch server when it is SSL protected.

A sample Kibana configuration is as follows:

apigw:
kibana:
dashboardInstance: http://localhost:9405
autostart: true
elasticsearch:
sslCA: C:/softwares/elasticsearch/config/SAG-B1HPWT2.pem
sslCert: C:/softwares/elasticsearch/config/SAG-B1HPWT2.crt
sslKey: C:/softwares/elasticsearch/config/SAG-B1HPWT2.key

Filebeat Configuration

The Filebeat configuration supports configuring the SSL certificate related settings that are used to connect to the SSL protected Elasticsearch server. The key configurations are as follows:

sslCA. A list of paths to the PEM file for the certificate authority for the Elasticsearch instance.

sslCert. The path to the PEM format certificate for SSL client authentication.

sslKey. The client certificate key used for client authentication. These files are used to verify the identity of Kibana to Elasticsearch server when it is SSL protected.

A sample Filebeat configuration is as follows:

apigw:
filebeat:
output:
elasticsearch:
sslCA: C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.pem
sslCert: C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.crt
sslKey: C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.key

Cluster Configuration

Note:
Install and run Terracotta server (a version that is compatible with API Gateway 10.5) for clustering API Gateway instances.

The cluster configuration contains the Terracotta clustering as well as Elasticsearch clustering settings. The key configurations are as follows:

aware, name, tsaUrls, sessTimeout, actionOnStartupError. All are Terracotta cluster settings that are set in the server watt properties.

terracottaLicenseFileName. The Terracotta server license file name. The file should be present in the folder SAGInstallDir/common/conf. API Gateway would use this file for joining the Terracotta cluster.

esClusterName. The Elasticsearch cluster name.

discoverySeedHosts. The discovery.seed_hosts property in the elasticsearch.yml file of Internal data store.

initialMasterNodes. The cluster.initial_master_nodes property in the elasticsearch.yml file of API Gateway data store.

A sample Cluster configuration is as follows:

apigw:
cluster:
aware: true
name: APIGatewayTSAcluster
tsaUrls: daeirnd33974:9510
terracottaLicenseFileName: terracotta-license.key
sessTimeout: 20
actionOnStartupError: standalone
esClusterName: SAG_EventDataStore
discoverySeedHosts: daeirnd33974:9340,daeirnd33562:9300,daeirnd33974:8300
initialMasterNodes : daeirnd33974_master

For terracottaLicenseFileName property a valid license file should be present in the SAGInstallDir/common/conf location, otherwise the property is ignored.

Note:
When cluster settings are given in the configuration files, the API Gateway server, on startup, would update the internal settings with the values from the configuration files but the node does not join the cluster. You have to restart the server for the cluster settings to become effective and for the node to join the cluster.

Consolidating Externalized Configuration Files

You can consolidate the configurations of different inter-components and cluster in a single configuration file.

A sample consolidated configuration file is as follows:

apigw:
elasticsearch:
tenantId: "apigateway"
hosts: "localhost:9240"
autostart: "true"
http:
username: ""
password: "@secure.elasticsearch.http.password"
keepAlive: "true"
keepAliveMaxConnections: 10
keepAliveMaxConnectionsPerRoute: 100
connectionTimeout: 1000
socketTimeout: 10000
maxRetryTimeout: 100000
https:
enabled: "false"
truststoreFilepath: "C:/softwares/elasticsearch-version/config/truststore-new.ks"
keystoreAlias: "root-ca"
truststorePassword: "@secure.elasticsearch.http.truststore.password"
enforceHostnameVerification: "false"
sniff:
enable: "false"
timeInterval: 1000
outboundProxy:
enabled: "false"
alias: "esoutboundproxyalias"
clientHttpResponseSize: 1001231
kibana:
dashboardInstance: "http://localhost:9405"
autostart: "true"
elasticsearch:
sslCA: "C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.pem"
filebeat:
output:
elasticsearch:
sslCA: "C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.pem"
cluster:
aware: "true"
name: "APIGatewayTSAcluster"
tsaUrls: "VMYAI105BVT06:9510"
terracottaLicenseFileName: "terracotta-license.key"
sessTimeout: "20"
actionOnStartupError: "standalone"
esClusterName: "SAG_EventDataStore"
discoverySeedHosts: "daeirnd33974:9340,daeirnd33562:9300,daeirnd33974:8300"
initialMasterNodes: apigw1_master

Similarly, you consolidate separate property files into a single file as shown in the following sample.

apigw.elasticsearch.tenantId=apigateway
apigw.elasticsearch.autostart=true
apigw.elasticsearch.hosts=localhost:9240
apigw.elasticsearch.clientHttpResponseSize=1001231
apigw.elasticsearch.http.keepAlive=true
.
.
.
apigw.kibana.dashboardInstance=http://localhost:9405
apigw.kibana.elasticsearch.sslCert=/path/to/your/client.crt
apigw.kibana.elasticsearch.sslKey=/path/to/your/client.key
apigw.kibana.elasticsearch.sslCA=C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.pem
.
.
.
.
apigw.filebeat.output.elasticsearch.sslCert=/path/to/your/client.crt
apigw.filebeat.output.elasticsearch.sslKey=/path/to/your/client.key
apigw.filebeat.output.elasticsearch.sslCA=C:/softwares/elasticsearch-version/config/SAG-B1HPWT2.pem
.
.
.
apigw.cluster.tsaUrls=daeirnd33974:9510
apigw.cluster.actionOnStartupError=standalone
apigw.cluster.discoverySeedHosts=daeirnd33974:9340,daeirnd33562:9300,daeirnd33974:8300
apigw.cluster.esClusterName=SAG_EventDataStore
apigw.cluster.name=APIGatewayTSAcluster
apigw.cluster.sessTimeout=20
apigw.cluster.terracottaLicenseFileName=terracotta-license.key
apigw.cluster.initialMasterNodes= daeirnd33974_master

Master configuration YAML file and its usage

The master configuration file, config-sources.yml, contains the paths, metadata, and properties for the other configuration files. The master configuration file and the other configuration files should be present in the folder SAGInstallDir\IntegrationServer\instances\instance_name\packages\WmAPIGateway\resources\configuration. The master configuration file can contain references to both YAML and Properties file types.

The master configuration file is read by API Gateway on startup. Using this file API Gateway reads the different configurations provided in the folder. If any entry has an invalid file name or path it is ignored but the error is logged into the API Gateway logs.

A sample master configuration file is as follows:

######################## Master configuration ############################

# This is the master configuration file which contains the configuration
# source definitions.
#

#======================= Sources configuration ==========================

#sources:

#--------------------- YAML file configuration source --------------------

- type: YAML
allowEdit: true
properties:
location: allExternal-settings.yml

#------------------- Properties file configuration source ----------------

#- type: PROPERTIES
# allowEdit: true
# properties:
# location: system-settings.properties
#

#================================== END =================================

The table lists and explains the properties of a configuration file source entry.

Property	Detail
type	Indicates the type of the configuration source. The applicable types are YAML, PROPERTIES and CC_YAML. YAML. A YAML configuration file. PROPERTIES. A properties configuration file. CC_YAML. A YAML configuration file, which is reserved for Command Central updates.
allowEdit	Indicates whether this file can be updated from API Gateway and is useful for hiding passwords. Valid values are true and false. If the value is set to true, it hides the clear text passwords. If the value is set to false, it displays the clear text passwords.
properties	Properties that enable API Gateway to connect to the defined configuration source. For the 10.5 release only the location property is supported. location. An absolute or relative path to a component-specific configuration file. In case of relative path, the file would be located relative to the system-defined location SAGInstallDir\IntegrationServer\instances\ instance_name\packages\WmAPIGateway\resources\configuration. Important: For the CC_YAML file type, the location is fixed as cc-config.yml. This file must not be modified manually as it is updated directly by Command Central. Instead, use the Command Central interfaces to modify this file.

Note:
The master configuration filename config-sources.yml is system-defined. A file with a different name is not treated as the master configuration file.

Hiding Clear Text Passwords in Configuration Files

To prevent unauthorized users from reading the credentials in the configuration files and other potential threats, the Administrator can enable hiding of such secrets by setting the allowEdit flag to true in the master configuration file. When allowEdit is set to true the secret values in the configuration files are stored in the Password manager and the plain text values in the files are replaced with the Password manager keys on API Gateway startup. After this, a user can see only the password keys in the files. On startup, API Gateway would retrieve the passwords for those settings from the Password manager using those keys and hence it is advised not to alter any of the password manager key values in the file. The passwords can be modified at any time and the same are replaced with the Password manager keys in the next API Gateway startup.

This table provides the list of the settings and their respective Password manager keys.

Setting	Password manager key replacement
apigw: elasticsearch: http: username: elastic	@secure.elasticsearch.http.password
apigw: elasticsearch: https: keystorePassword: 6572b9b06156a0ff778c	@secure.elasticsearch.http.keystore.password
apigw: elasticsearch: https: truststorePassword: 6572b9b06156a0ff778c	@secure.elasticsearch.http.truststore.password

Properties File Support for Externalized Configurations

In addition to YAML files, configurations can be saved in Properties files as well. The property names are the same as those in the YAML configuration files. The property names in Properties files are delimited by a "." for forming the property name. For example. the tenantId property under apigw > elasticsearch in YAML, can be specified as apigw.elasticsearch.tenantId in the properties file.

A sample Properties file is as follows:

apigw.elasticsearch.tenantId=default
apigw.elasticsearch.autostart=false
apigw.elasticsearch.hosts=vmabc\:9240
apigw.elasticsearch.http.password=admin123
apigw.elasticsearch.http.username=admin
apigw.kibana.dashboardInstance=http://localhost:9405
apigw.kibana.elasticsearch.sslCert=/path/to/your/client.crt

Configuring Multiple Configuration Files and Its Effects

The master configuration file can have many entries (0 to N) for defining multiple configuration files as configuration sources. When such a file is used to start API Gateway, the configuration values from all the files would be merged into a single effective configuration. If the same configuration value is present in two files, then the value in the file which has a higher preference is given priority. The order of preference is in the reverse order in which they are defined in the master configuration file, that is, the configuration values that are defined in the last configuration file entry would have the highest preference. A sample use case is explained below.

Assume file1.yml has the following configurations.

apigw:
elasticsearch:
tenantId: default

And, file2.properties has the following configurations.

apigw.elasticsearch.tenantId=apigateway

And, file3.yml has the following configurations.

apigw:
elasticsearch:
http:
username: admin
password: admin123
kibana:
dashboardInstance: http://localhost:5601

Then the combined configuration that becomes effective is as follows.

Effective config.yml configuration:

apigw:
elasticsearch:
tenantId: apigateway
http:
username: admin
password: admin123
kibana:
dashboardInstance: http://localhost:5601

Limitations

Currently, externalized configurations are limited to inter-component and cluster configurations.

If you have defined cluster configuration in the externalized configuration file, on startup the API Gateway server updates the internal settings with the values from the externalized configuration files but the node in the cluster will not be updated. API Gateway server restart is required for the cluster settings to become effective and to join the cluster.