Troubleshooting Tips: Securing API Data Store (Elasticsearch)

I see an error with API Data Store internal communication after configuring SSL with Search Guard plugin in a cluster setup

The following error message appears when there is a problem with the node certificate:

Caused by: org.elasticsearch.ElasticsearchException: bad header found. This means typically that one node try to connect to another with a non-node certificate (no OID or searchguard.nodes_dn incorrect configured) or that someone is spoofing requests. See https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

For communication with API Data Store:

Generate a node certificate and ensure that you provide a proper subject name in the certificate.

Add the same subject name from the node certificate that you have generated to the following field in elasticsearch.yml file located at SAGInstallDir\InternalDataStore\config folder for all the nodes.

Note:
Client certificate cannot be used for node-based authentication.

See How do I Secure API Data Store Communication using HTTPS with Search Guard Plugin ? to configure Search Guard plugin.

I see that API Data Store fails to start after securing with Search Guard plugin

The following error message appears when searchguard.nodes_dn property is configured with a value other than the Common Name (CN) values in elasticsearch.yml file located at SAG_Install_Directory\InternalDataStore\config folder:

Root cause: MasterNotDiscoveredException[null]

(org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException)

Modify the value of searchguard.nodes_dn with the actual CN values in elasticsearch.yml file located at SAGInstallDirectory\InternalDataStore\config folder and save the file.

I see that API Gateway fails to connect to API Data Store after it is secured with SSL using Search Guard plugin

The following error message appears when API Gateway fails to connect to API Data Store after securing with Search Guard plugin:

[ERROR][c.f.s.h.SearchGuardHttpServerTransport] [nlxdsmcv84.srvfarm4-eur.dsm-group.com1560510563040] SSL Problem Received fatal alert: certificate_unknown javax.net.ssl.SSLException: Received fatal alert: certificate_unknown.

2. Rename the folder search-guard-5 with search-guard-ssl located at SAGInstallDirectory\InternalDataStore\plugins.

3. Check if API Data Store HTTPS connectivity is properly configured in the following location: SAGInstallDirectory\IntegrationServer\instances\instance_name\packages\WmAPIGateway\config\resources\elasticsearch\config.properties.

I see an error while securing API Data Store using Search Guard plugin

The following error message appears while securing API Data Store using Search Guard plugin:

Caused by: java.security.cert.CertificateExpiredException: NotAfter:

This problem might occur when your certificate is expired.

Provide a valid node-0-keystore.jks and truststore.jks certificate while securing API Data Store with Search Guard plugin.

I see an error while accessing API Data Store when it is secured using Search Guard plugin

A sample error message is as follows:

{ "error":{
"root_cause":[
{ "type":"security_exception",
"reason":"no permissions for [] and User [name=Administrator, roles=[admin],
requestedTenant=null]"
}
],
"type":"security_exception",
"reason":"no permissions for [] and User [name=Administrator, roles=[admin],
requestedTenant=null]"
},
"status":403
}

Run the following script with the client certificate with Common Name (CN) value configured in searchguard.authcz.admin_dn parameter in elasticsearch.yml located at SAG_Install_Directory\InternalDataStore\config.

curl -L -X PUT 'https://xxxx:9240/_all/_settings' \
-H 'Authorization: Basic QWRtaW5pc3RyYXRvcjpCIXJ1cEUkQGRtaJu7' \
-H 'Content-Type: application/json' \
--cert <public key file>

Note:
If you use jks stores for the Search Guard configuration, extract the private and public key using the following keytool https://security.stackexchange.com/questions/3779/how-can-i-export-my-private-key-from-a-java-keytool-keystore.

During authentication a pop-up window appears when I try to view any dashboards in the Analytics tab

At times, when I view the Analytics tab, the Authentication pop-up window appears. This might be because Kibana is secured with SSL and API Gateway is unable to connect to Kibana. As of now, API Gateway does not support enabling SSL for Kibana. Kibana ports can be blocked from external access through firewall configuration.

Remove the following properties from kibana.yml file located at SAGInstallDir\profiles\IS_default\apigateway\dashboard\config:

server.ssl.cert: "/eip/apps/sag/InternalDataStore/config/PVWSLDWM001_pem.cer"

server.ssl.key: "/eip/apps/sag/InternalDataStore/config/PVWSLDWM001_pem.key"

I get an error message while securing API Data Store communication using HTTPS

If the UI takes sometime a longer duration to load, navigate to the following location and set the value of the property xpack.fleet.enabled to false.SAG_Install_Dir/wm10.11/profiles/IS_apigateway/apigateway/dashboard/config/kibana.yml

Note:
Ignore the following error message - Failed to gather field mappings for index [metrics-endpoint.metadata_current_default] org.elasticsearch.action.NoShardAvailableActionException: No shard available for [org.elasticsearch.action.fieldcaps.FieldCapabilitiesIndexRequest/unset].