Secure Sockets Layer (SSL) in System Management Hub


Overview of SSL

Originally developed by Netscape, the Secure Sockets Layer (SSL) protocol has been universally accepted on the Internet for authenticated and encrypted communication between clients and servers. It is included as part of both Microsoft and Netscape browsers and most web servers.

SSL protects data transferred over HTTP using encryption enabled by a server’s SSL certificate. The SSL certificate contains unique authenticated information about the certificate’s owner and verifies his or her identity.

An SSL certificate is a digital document that consists of a digital signature to bind together a public key with an identity - information such as the name of a person, an organization, an address, etc. A public key is used to encrypt information and a private key is used to decipher it. When a browser points to a secured domain, an SSL handshake authenticates the server and the client and establishes an encryption method, and a unique session key.

A trust store must be created to keep the certificates that the other party trusts to verify messages. These certificates (or trust store entries) contain the keystores that the client and server must have during the SSL handshake.

The keystore is like a container for the public and private keys that the application using SSL protocol utilizes along with additional information in order to sign the messages.

Server and Client Configurations

Usually, this is done by default. All web browsers use SSL over HTTP.

Encryption Algorithms (Ciphers)

The client and the server use the same algorithms to encrypt all the communication during the session. SSL protocol uses a combination of public keys and symmetric keys for data encryption. Symmetric key encryption is much faster than public-key encryption, but public-key encryption provides better authentication techniques.

SSL protocol supports the use of different cryptographic algorithms (or ciphers) for authenticating the server and the client to each other, transmitting certificates and establishing session keys. Clients and servers may support different sets of ciphers depending on the SSL version and the company policies for encryption strength.

SSL Handshake

The SSL handshake protocol determines how the server and the client negotiate the sets of ciphers that they use to authenticate each other. The result of this authentication is the transmitting of certificates and establishing session keys between them.

An SSL session always begins with an exchange of messages called the SSL handshake. The handshake tampers detection during the authentication session and does the following actions:

  • The server authenticates itself to the client using public-key techniques

  • The client and the server cooperate in the creation of symmetric keys used for rapid encryption (and decryption)

  • (optional) Authentication of the client to the server

SSL Limitations

Following are several of the limitations of SSL:

  • Latency

    Latency is increased with the SSL connection since there are additional round trips added in order to establish authentication. This is only a fraction of a second for each connection, so it is not an issue on most small or medium businesses. For large business processing, however (with numerous transactions per minute) this can add up, especially if a client authentication is required.

  • Bandwidth

    Bandwidth considerations are not an issue for most small- or medium-sized businesses. SSL transactions can, however, increase by about 1 KB each. This is an issue with large numbers of transactions.

  • Processor Usage

    Processor usage is increased with SSL connections. This is an issue only on servers running numerous transactions per minute. Processor usage can also be minimized by using the most processor-efficient encryption methods.

Registry Entries for Windows

[HKEY_LOCAL_MACHINE\SOFTWARE\Software AG\System Management Hub\Global]

"PKCS12File"="{{ARGDIR}}\\files\\server.p12"
"SSL"="0"
"SSL_Server_Cnf"="{{ARGDIR}}\\files\\ssl\\server.cnf"

{{ARGDIR}} is the System Management Hub installation directory. By default, it is the Software AG_directory\InstanceManager directory.

Registry Entries for UNIX

[HKEY_LOCAL_MACHINE\SOFTWARE\Software AG\System Management Hub\Global]

"PKCS12File"="{{ARGDIR}}/files/server.p12"
"SSL"="0"
"SSL_Server_Cnf"="{{ARGDIR}}/files/ssl/server.cnf"

{{ARGDIR}} is the System Management Hub installation directory. By default, it is Software AG_directory\InstanceManager.

SSL Workflow

There are two separate connections that have to be secured:

  • The connection between the client server and the proxy

  • The connection between the proxy and the target machine

System Management Hub provides you with a secure SSL-encrypted connection only between the proxy and the target system. If the SSL configuration on the web server is not enabled, the connection between the client server (web or batch interface) and the proxy is not secured.

graphics/ssl_security.png

To have a secure connection from the proxy system to a managed host, you must enable the SSL functionality and configure System Management Hub to either create or upload an SSL certificate.

For more details on how to do that, see Enabling SSL Support in System Management Hub.

Note:
In order to use Secure Sockets Layer (SSL) between the client server and the proxy, you must configure Software AG Runtime to use SSL.

Troubleshooting Guidelines

Following is a list with the possible configuration problems:

  • Missing or wrong registry entry for PKCS12 keystore location

  • Internal error in the SSL engine

  • Missing private key file part of the keystore

  • Wrong PKCS12 password

  • Extraction of the certificate from the keystore is not possible

  • Extraction of the private key from the keystore is not possible

The SSL message on the following screen capture registers problems with your host' SSL configuration:

graphics/ssl_alert.png

If you select Yes, you continue the session without encryption of the information transmitted between MIL and the Client/Server Layer. The system remembers your choice, and it does not notify you again if the same problem occurs.

If you select No, you cannot run commands in batch interface. Ignore this error message by choosing Yes.

For information on how to ignore SSL error messages in batch interface, see Batch Commands.

To diagnose the exact cause of the problem, you must activate the trace log function. Access the registry master key (HKEY_LOCAL_MACHINE\\SOFTWARE\\Software AG\\System Management Hub) and change the settings of two of the customizable keys of the Client/Server Layer. They are disabled ("0") by default.

Key Description Settings
Trace String;

Controls trace file.

  • "0" disabled (default);

  • "1" enable trace, write trace data to file (defined by user);

  • "2" enable trace, write trace data to STDERR;

  • "3" enable trace, write trace data to BOTH file and STDERR.

Trace_Byte_Level String;

Controls trace of TCP/IP and IPC communication, active only if Trace is enabled.

  • "0" disabled (default);

  • "1" enable.

Switch Trace and Trace_Byte_Level keys to "1" (enabled) in your Client/Server Layer registry branch.

The trace log path parameter is Agent_stderr directory. This directory defines the location where agent trace (if enabled) is written.

Following is a sample of a log file with an error message for a faulty SSL configuration:

15:38:59.609 (2320)NET DRIVES: ARGUSUtlNetEumerateDrives: ret TRUE
15:38:59.625 (2320)SSL error: Can't get a PKCS12 certificate
15:39:02.406 (2320)Flow : Thread started: 1

Error Messages

Error message Description
Empty Public Certificate File The certificate file part of the keystore is missing.
Empty Private Key File The private key file part of the keystore is missing.
SSL_CTX_use_certificate This prefix points to internal error in SSL engine.
SSL_CTX_use_PrivateKey This prefix points to internal error in SSL engine.
PKCS12 File entry is missing The registry entry for PKCS12 keystore location is wrong or missing.
Can't get the PKCS12 password The PKCS12 password is wrong.
Can't get the PKCS12 certificate It is not possible to extract the certificate from the keystore.
Can't get the PKCS12 private key It is not possible to extract the private key from the keystore.