Security Core Concepts
Security features in Terracotta
Terracotta provides the following security features:
1. Connection encryption - encrypts connections between clients and servers, and also between servers, using the SSL/TLS protocol.
2. Cluster authentication - validates the identity of processes initiating connections to a server in a Terracotta cluster.
3. TMS authentication - validates the identity of users attempting to use the TMS.
4. TMS authorization - determines if an authenticated user has access to perform a given operation in TMS.
5. Auditing - writes security-relevant events to audit logs.
6. IP whitelisting - restricts access to only allow clients from known IP addresses.
Security Root Directory
To configure security features in Terracotta, each server and, in most cases, each client, must have a security root directory. The security root directory is a filesystem location for certificates, passwords and other security-related files.
The security root directory, and the files and subdirectories contained in it should be readable by the respective client or server process.
Note: Ensure that the security root directory is accessible only by users who are permitted to run the respective Terracotta client or server.
When a security root directory is specified, it can be given as absolute or relative. If specified in a tc-config.xml file, this is relative to the directory where the tc-config file is located.
Structure of the Security Root Directory
The directory structure below lists all possible subdirectories and files that can be present in a security root directory. Note that any one security root directory will have only a subset of these.
<security-root-directory>
├── access-control
│ ├── ldap.properties
│ └── users.xml
├── identity
│ ├── credentials.properties
│ └── <common-name>-<timestamp>.jks
├── trusted-authority
│ └── trusted-authority-<timestamp>.jks
└── whitelist.txt
Note: It is recommended to keep only the required files and directories under the security root directory to prevent configuration errors. If any unidentifiable files or directories are found, an appropriate warning message will be logged.
Access control subdirectory
The access-control subdirectory should only be specified on the server-side. It contains files to configure authentication and authorization:
ldap.properties - configuration properties for using an LDAP server for authentication or authorization
users.xml - used for file-based authentication or authorization
Identity subdirectory
The identity subdirectory can be specified on both the client-side and the server-side. It contains certificates or password-based credentials to prove the identity of the process.
credentials.properties - a standard Java properties file containing two properties: username and password. These credentials are sent to the server when the connection is established, in order to authenticate. This file should be present when file-based authentication or LDAP-based authentication is configured.
<common-name>-<timestamp>.jks - a keystore containing a certificate for proving identity. Note that the <common-name> part of the filename must match the Common Name specified in the certificate and the <timestamp> represents the time that the certificate was created.
Trusted authority subdirectory
The trusted-authority subdirectory can be specified both client-side and server-side. It contains trusted root certificates.
trusted-authority-<timestamp>.jks - a truststore containing a trusted root certificate for validating identity certificates. Note that the <timestamp> represents the time that the certificate was created.
The whitelist.txt file
The whitelist.txt file should only be specified on the server-side. It contains details of client IP addresses permitted to establish connections with the cluster.
Certificates
This section assumes a good understanding of SSL/TLS fundamentals.
Certificate creation
Note: Certificates must be created using the RSA algorithm, preferably with a key size of 4096.
The keystores and truststores, that are deployed to the identity and trusted-authority directories respectively, can be created in any way desirable as long as the following rules are followed:
Keystore creation
Keystore must be of type
jks.
Keystore filename must be in
${common name}-${yyyyMMddThhmmss}.jks format (e.g.
com.organization.host-20180223T102319.jks).
yyyyMMddThhmmss should represent the time of creation (timestamp) of the file. When multiple keystores are present, the keystore with the latest timestamp is used.
Keystore must have only one
terracotta_security_alias entry, and it should contain the identity certificate and the corresponding private key.
Common Name field in the Distinguished Name in the certificate must match the common name fragment in the keystore filename. For a server the common name must match the host name.
The identity certificate must be within its period of validity.
The password for the keystore and the
terracotta_security_alias store entry must be
terracotta_security_password.
The certificate must be created using the RSA algorithm, preferably with a key size of 4096.
Truststore creation
Truststore must be of type "jks".
Truststore filename must be in
trusted-authority-${yyyyMMddThhmmss}.jks format (e.g.
trusted-authority-20180223T102319.jks). When multiple truststores are present, all of them are used irrespective of their timestamps.
Truststore must have only one
terracotta_security_alias entry, and it should contain a trusted certificate.
The trusted certificate must be within its period of validity.
The password for the truststore must be
terracotta_security_password.
The certificate must be created using the RSA algorithm, preferably with a key size of 4096.
Certificate rotation
If the certificates expire or get compromised, they must be rotated. Following are the different ways of rotating them:
Certificate rotation with cluster shutdown
The followed steps need to be performed in order:
1. Generate new keystore and truststore files following the rules mentioned in the Keystore creation and Truststore creation sections above.
2. Shut down all the servers and clients.
3. Replace the old keystores and truststores with the new keystores and truststores in the corresponding identity and trusted-authority directories of each client and server.
4. Start all the servers and the clients for the new certificates to take effect.
The above sequence has the advantage that it is simple to perform, with the drawback of requiring the entire cluster and the clients to be restarted.
Certificate rotation with rolling restarts
The followed steps need to be performed in order:
1. Generate new keystore and truststore files following the rules mentioned in the Keystore creation and Truststore creation sections above.
2. Deploy new truststores in corresponding trusted-authority directories of each client and server.
3. Restart all the passive servers and clients. Once the passive servers reach PASSIVE-STANDBY status, restart all the active servers.
4. Replace old keystores with the new keystores in corresponding identity directories of each client and server.
5. Restart all the passive servers and clients. Once the passive servers reach PASSIVE-STANDBY status, restart all the active servers.
6. Delete old truststores from corresponding trusted-authority directories of each client and server.
7. Restart all the passive servers and clients. Once the passive servers reach PASSIVE-STANDBY status, restart all the active servers.
The above sequence has the advantage that it does not require cluster downtime, with the drawback of having a significant number of steps.
Auditing
Terracotta can audit when security-relevant events occur. You can configure this by specifying an audit directory.
Note: Ensure that the audit directory is accessible only by users who are permitted to run the respective Terracotta server or by users who are allowed to read the audit log.
Audit directory structure
The auditing process creates a hierarchical filesystem structure under the audit directory:
When an auditable event occurs, an appropriate message is written to the audit log file. Each line in the audit log corresponds to a single event.
Audit log files generated by a server in a Terracotta cluster are be stored under the server directory. Audit log files generated by the TMS are be stored under the tmc directory.
Each server has a separate directory named after the hostname of the server and the port number on which the server process is listening. This allows multiple servers to use the same audit directory.
Under the server-specific directory, logs are organised by date. If any audit log file exceeds 10 MB, a new audit log file is created.
Audit entry sanitization
Any dynamic information, such as the username used in a login attempt, is sanitized before being logged.
If the sanitization process finds no unacceptable characters or character sequences, then the information is logged surrounded with backticks.
For example, if the user alex successfully authenticated, that is audited as:
LDAP authentication success:- IP: `203.0.113.1`, User: `alex`
However, if the santization process finds unacceptable characters or character sequences, then the information is logged surrounded with tildes and prefixed with sanitized. Characters replaced in the sanitization process appear in the format |U+xxxx| where the xxxx is the unicode codepoint in hexadecimal.
Note that there may be more than four digits in the hex codepoint due to supplementary unicode characters.
For example, if the user jürgen successfully authenticated, that is audited as:
LDAP authentication success:- IP: `203.0.113.1`, User: ~sanitized j|U+00fc|rgen~
If a string exceeds 2000 characters, then it will be truncated and a * added to the end to indicate the truncation.
~, |, *, ` and + are all treated as unacceptable characters and sanitized. Other strings sequences are also sanitized, particularly any characters outside the standard, printable ASCII range.