SSL / TLS Troubleshooting guide
This document provides a list of the most commonly seen problems related to
Cluster Security, and their solutions:
Problem category: Host fails to start
This section describes the most commonly seen problems related to a host (server or a client) startup.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
configured security-root-directory /path/to/security-root-directory
does not exist
Diagnosis
The specified security root directory does not exist.
Action
Make sure that the directory exists and contains identity and trusted-authority directories with valid keystores and truststores in them.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
identity directory doesn't exist in configured
security-root-directory /path/to/security-root-directory
Diagnosis
The specified security root directory does not contain an identity directory.
Action
Make sure that the directory exists inside the security root directory and contains valid keystores.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
trusted-authority directory doesn't exist in configured
security-root-directory /path/to/security-root-directory
Diagnosis
The specified security root directory does not contain a trusted-authority directory.
Action
Make sure that the directory exists inside the security root directory and contains valid truststores.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
No acceptable keystore files found in identity directory
/path/to/security-root-directory/identity
Diagnosis
Either of:
identity directory does not contain any keystores.
identity directory contains keystores, but their file names are not in the format
${common name}-${yyyyMMddThhmmss}.jks (e.g.
com.organization.host-20180223T102319.jks).
Action
Make sure that
identity directory contains keystores which follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
No acceptable truststore files found in trusted-authority directory
/path/to/security-root-directory/trusted-authority
Diagnosis
Either of:
trusted-authority directory does not contain any truststores.
trusted-authority directory contains truststores, but their file names are not in the format
${common name}-${yyyyMMddThhmmss}.jks (e.g.
trusted-authority-20180223T102319.jks).
Action
Make sure that
trusted-authority directory contains truststores which follow the truststore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
Tried to use the password terracotta_security_password to load the
keystore file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
but that failed
Diagnosis
Latest keystore file does not have terracotta_security_password as its password, where latest keystore file is the keystore file with the latest timestamp string in the filename (e.g., host-20180131T120830.jks is considered newer than both host-20170131T120830.jks and host-20180131T120822.jks).
Action
Make sure the keystores follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
Tried to use the password terracotta_security_password to read the
keystore entry with alias terracotta_security_alias in the keystore
file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
but that failed
Diagnosis
Latest keystore file does not have terracotta_security_password as terracotta_security_alias entry password, where latest keystore file is the keystore file with the latest timestamp string in the filename (e.g., host-20180131T120830.jks is considered newer than both host-20170131T120830.jks and host-20180131T120822.jks).
Action
Make sure the keystores follow the follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
Unable to find required private key/certificate chain entry using
alias terracotta_security_alias in keystore file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
Diagnosis
Latest keystore file does not have terracotta_security_alias as certificate alias, where latest keystore file is the keystore file with the latest timestamp string in the filename (e.g., host-20180131T120830.jks is considered newer than both host-20170131T120830.jks and host-20180131T120822.jks).
Action
Make sure the keystores follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
Certificate in keystore file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
is expired
Diagnosis
Latest keystore file contains an expired certificate, where latest keystore file is the keystore file with the latest timestamp string in the filename (e.g., host-20180131T120830.jks is considered newer than both host-20170131T120830.jks and host-20180131T120822.jks).
Action
Make sure the keystores follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
Certificate in keystore file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
is not valid yet
Diagnosis
Latest keystore file contains a certificate with a future start date, where latest keystore file is the keystore file with the latest timestamp string in the filename (e.g., host-20180131T120830.jks is considered newer than both host-20170131T120830.jks and host-20180131T120822.jks).
Action
Make sure the keystores follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
The common name org.host of the certificate that was loaded from
keystore file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
doesn't match the common name com.organization.host in the filename
com.organization.host-20180131T120830.jks
Diagnosis
Common Name field in the Distinguished Name in the host's certificate does not match the common name fragment in the latest keystore filename.
Action
Make sure the keystores follow the keystore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
No valid trusted certificates found in trusted-authority directory
/path/to/security-root-directory/trusted-authority; Unable to find
required trusted certificate entry using alias
terracotta_security_alias in truststore file
/path/to/security-root-directory/trusted-authority/trusted-authority-20180131T120832.jks
Diagnosis
No valid truststores were found. The specific truststore reported in the Exception message contains a certificate which uses an alias other than terracotta_security_alias. Note that this Exception can be followed by one or more Suppressed Exceptions that can indicate why other truststores could not be used.
Action
Make sure the truststores follow the truststore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
No valid trusted certificates found in trusted-authority directory
/path/to/security-root-directory/trusted-authority; Certificate in
truststore file
/path/to/security-root-directory/trusted-authority/trusted-authority-20180131T120834.jks
is expired
Diagnosis
No valid truststores were found. The specific truststore reported in the Exception message contains an expired certificate. Note that this Exception can be followed by one or more Suppressed Exceptions that can indicate why other truststores could not be used.
Action
Make sure the truststores follow the truststore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
No valid trusted certificates found in trusted-authority directory
/path/to/security-root-directory/trusted-authority; Certificate in
truststore file
/path/to/security-root-directory/trusted-authority/trusted-authority-20180131T120834.jks
is not valid yet
Diagnosis
No valid truststores were found. The specific truststore reported in the Exception message contains a certificate with a future start date. Note that this Exception can be followed by one or more Suppressed Exceptions that can indicate why other truststores could not be used.
Action
Make sure the truststores follow the truststore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
No valid trusted certificates found in trusted-authority directory
security-root-client\trusted-authority; Tried to use the password
terracotta_security_password to load the truststore file
security-root-client\trusted-authority\trusted-authority-20180131T120832.jks
but that failed
Diagnosis
No valid truststores were found. The specific truststore reported in the Exception message does not have terracotta_security_password as its password. Note that this Exception can be followed by one or more Suppressed Exceptions that can indicate why other truststores could not be used.
Action
Make sure the truststores follow the truststore creation rules as described in the section
Certificates.
Symptom
Host fails to start with an Exception message similar to:
java.lang.RuntimeException:
com.terracottatech.security.common.exception.SecurityConfigurationException:
Unable to validate certificate chain with alias
terracotta_security_alias in keystore file
/path/to/security-root-directory/identity/com.organization.host-20180131T120830.jks
using truststore file(s)
Diagnosis
Host certificate in latest keystore file is not signed by any of the known trusted authorities, and thus cannot be validated by any of the truststore files. Latest keystore file here is the keystore file with the latest timestamp string in the filename (e.g., host-20180131T120830.jks is considered newer than both host-20170131T120830.jks and host-20180131T120822.jks).
Action
Make sure that the latest keystore file contained in the identity directory is signed by the truststores in the trusted-authority directory.
Problem category: Connection fails to establish
Symptom
org.terracotta.connection.ConnectionException:
com.terracottatech.connection.ProbableSecurityConfigurationException:
Handshake with server failed when this client tried to initiate a
non-secure connection. Possible reason: server is running with
security enabled.
Diagnosis
The client which tried to establish a connection to a server running with SSL/TLS configuration is not using an SSL/TLS configuration.
Action
Make sure the client uses a correct SSL/TLS configuration (via a secure API, an XML config, command parameters etc.) so that it can establish a secure connection with an SSL/TLS security-enabled server.
Symptom
org.terracotta.connection.ConnectionException:
com.terracottatech.connection.ProbableSecurityConfigurationException:
Handshake with server failed when this client tried to initiate a
secure connection. Possible reasons: client security configuration
is not valid, or server is not running with security
enabled.
Diagnosis
Either of:
1. The client is using an SSL/TLS security configuration but the server is not.
2. The client cannot validate the server because the server's CA certificate is not present in the client's trusted certificates.
3. The server cannot validate the client because the client's CA certificate is not present in the server's trusted certificates.
Action
Make sure that:
1. The client uses an unsecured configuration (via an unsecured API, an XML config, command parameters etc.) if the server is running with an unsecured configuration.
2. The client and the server certificates are signed by the same CA and their trusted-authority directories contain the same truststores.