Setting up Broker Instances

This document contains information on setting up the Broker under z/OS. It assumes that you have completed the relevant steps described under Installing EntireX under z/OS. It covers the following topics:


Setting up TCP/IP Transport

The recommended way to set up the TCP/IP communicator is to define PORT=nnnn and optionally HOST=x.x.x.x|hostname and STACK-NAME=stackname under TCP/IP-specific Attributes.

However, if no port number is specified in the broker attribute file, the EntireX Broker kernel uses getservbyname to determine the TCP/IP port on which it will listen for incoming connections. The specified name is the value of BROKER-ID in the attribute file. An entry for this value must be made in the local machine's /etc/services file. Example:

ETBnnn yyyyy/tcp # local host

where etbnnn is the BROKER-ID and yyyyy is the intended port number. This is the same place from which local Broker stubs will obtain the port information. If getservbyname fails, then a default port number of 1971 will be used. This is the same default port number that the stubs use.

Running Broker with SSL/TLS Transport

This section covers the following topics:

Introduction

The Broker can use Secure Sockets Layer/Transport Layer Security (SSL/TLS) as the transport medium. The term "SSL" in this section refers to both SSL and TLS. RPC-based clients and servers as well as ACI clients and servers are always SSL clients. The broker is always the SSL server. For an introduction see SSL/TLS, HTTP(S), and Certificates with EntireX in the platform-independent Administration documentation.

SSL delivered on a z/OS mainframe will typically use the Resource Access Control Facility (RACF) as the certificate authority (CA). Certificates managed by RACF can only be accessed through the RACF keyring container. A keyring is a collection of certificates that identify a networking trust relationship (also called a trust policy). In an SSL client/server network environment, entities identify themselves using digital certificates called through a keyring. Server applications on z/OS that wish to establish network connections to other entities can use keyrings and their certificate contents to determine the trustworthiness of the client or peer entity. Note that certificates can belong to more than one keyring, and you can assign different users to the same keyring. Because of the way RACF internally references certificates, they must be uniquely identifiable by owner and label, and also unique by serial number plus data set name (DSN).

For establishing an SSL connection on z/OS, IBM's Application Transparent Transport Layer Security (AT-TLS) can be used, where the establishment of the SSL connection is pushed down the stack into the TCP layer.

Using IBM's Application Transparent Transport Layer Security (AT-TLS)

Configure the AT-TLS rules for the policy agent (PAGENT) graphics/no4.gif using an appropriate client graphics/no1.gif and the z/OS Management Facility (z/OSMF) graphics/no2.gif. Together with SSL parameters (to provide certificates stored in z/OS as RACF keyrings) define AT-TLS rules, for example by using the application graphics/no5.gif job name and local TCP port number. If the rules match, the TCP connection is turned into an SSL connection graphics/no6.gif. Refer to your IBM documentation for more information, for example the IBM Redbook Communications Server for z/OS VxRy TCP/IP Implementation Volume 4: Security and Policy-Based Networking.

graphics/setupBroker_ssl.png

graphics/no1.gif Client to interact with z/OS Management Facility (z/OSMF).
graphics/no2.gif AT-TLS rules are defined with z/OSMF policy management.
graphics/no3.gif Policy Repository with AT-TLS rules stored as z/OS files.
graphics/no4.gif Policy Agent, MVS task PAGENT, provides AT-TLS rules through a policy enforcement point (PEP) to TCP/IP stack.
graphics/no5.gif Application using TCP connection.
graphics/no6.gif If AT-TLS rules match, the TCP connection is turned into an SSL connection.

Notes:

  1. The client graphics/no1.gif may vary per operating system, for example a Web browser for z/OS 2.1.
  2. z/OSMF graphics/no2.gif includes other administration and management tasks in addition to policy management.
  3. Policy Management graphics/no3.gif includes other rules, such as IP filtering, network address translation etc.

Start of instruction setTo set up SSL with AT-TLS

  1. To operate with SSL, certificates need to be provided and maintained. Depending on the platform, Software AG provides default certificates, but we strongly recommend that you create your own. See SSL/TLS Sample Certificates Delivered with EntireX in the EntireX Security documentation.

  2. Modify broker-specific attributes. Configure the Broker to use TCP:

    DEFAULTS = BROKER
      ...
      TRANSPORT = TCP
    
    DEFAULTS = TCP
      PORT = 1958

    See also Setting up TCP/IP Transport.

  3. Configure AT-TLS to turn the TCP/IP connection to an SSL connection, using a client to interact with the z/OS Management Facility (z/OSMF). The outcome of this configuration is a Policy Repository with AT-TLS rules stored as z/OS files. This file is the configuration file for the Policy Agent, MVS task PAGENT.

  4. Make sure the SSL clients connecting to the broker are prepared for SSL connections as well. See Using SSL/TLS with EntireX Components.

Migration from Broker's Direct SSL/TLS Support to AT-TLS

Steps

Start of instruction setTo migrate to AT-TLS

  1. Migrate Broker SSL port to TCP.

    Move the PORT=value line from the DEFAULTS=SSL section to the DEFAULTS=TCP section in the Broker attribute file to establish it as TCP/IP server port to be controlled by the Policy Agent PAGENT and TCP stack.

    After this step you can delete the DEFAULTS=SSL section in the Broker attribute file.

  2. Migrate Broker SSL attributes KEY-LABEL and TRUST-STORE.

    Configure AT-TLS to turn the TCP/IP connection to an SSL connection, using a client to interact with the z/OS Management Facility (z/OSMF). The outcome of this configuration is a Policy Repository with AT-TLS rules stored as z/OS files. This file is the configuration file for the Policy Agent, MVS task PAGENT.

    Part of this configuration is to provide the value of the Broker SSL attributes:

    • KEY-LABEL as value of parameter CertificateLabel in the policy statement TTLSConnectionAdvancedParms, for example CertificateLabel ExxAppCert.

    • TRUST-STORE as value of parameter Keyring in the policy statement TTLSKeyringParms, for example Keyring EXX/EXXRING.

z/OSMF Considerations

General steps to perform in the Network Configuration Assistant of the z/OS Management Facility:

  1. Create a new Traffic Descriptor and specify

    • the local port for an SSL server (Broker) or remote port for an SSL client (EntireX client or server)

    • the TCP connect direction (inbound for the Broker and outbound for an EntireX client or server)

    • the AT-TLS handshake role (server for the Broker or client for EntireX client or server)

    You can specify the jobname to define the connection to the Broker. You also need to define the existing RACF keyring (format user ID/keyring) or OMVS keystore.

  2. Expand or create a new Requirement Map to map the new Traffic Descriptor to an existing security level.

    Note:
    The client and server need to use the same security protocol.

  3. Create a new rule for the specific TCP/IP stack based on the Requirement Map. Install and activate the rule, using the PAGENT.

For more information on z/OSMF, see your IBM documentation.

Example

The following is an excerpt of a Policy Agent, MVS task PAGENT configuration file defining a Broker TCP/IP port as secure SSL port as outcome of this configuration:

TTLSRule                          ConnRule01~33
    {
      LocalAddr                       ALL
      RemoteAddr                      ALL
      LocalPortRangeRef               portR1
      RemotePortRangeRef              portR2
      Jobname                         <job_name> e.g. ETBNUC
      Direction                       Inbound
      Priority                        223
      TTLSConnectionActionRef         cAct1
      TTLSEnvironmentActionRef        eAct1
      TTLSGroupActionRef              gAct1
    }
    TTLSConnectionAction              cAct1
    {
      HandshakeRole                   Server
      TTLSCipherParmsRef              cipher1
      TTLSConnectionAdvancedParmsRef  cAdv1
      CtraceClearText                 Off
      Trace                           6
    }
    TTLSEnvironmentAction             eAct1
    {
      HandshakeRole                   Server
      EnvironmentUserInstance         0
      TTLSKeyringParmsRef             keyR1
    }
    TTLSGroupAction                   gAct1
    {
      TTLSEnabled                     On
      Trace                           6
    }
    TTLSConnectionAdvancedParms       cAdv1
    {
      CertificateLabel                <certificate_label> e.g. ExxAppCert
      SecondaryMap                    Off
      SSLv3                           Off
      TLSv1                           On
      TLSv1.1                         On
      TLSv1.2                         On
    }
    TTLSKeyringParms                  keyR1
    {
      Keyring                         <user_id / keyring> e.g. EXX/EXXRING
    }
    PortRange                         portR1
    {
      Port                            <port_number> e.g. 1958
    }
    PortRange                         portR2
    {
      Port                            1024-65535
    }
    CipherParms                       cipher1
    {
      V3CipherSuites                  TLS_RSA_WITH_AES_256_GCM_SHA384
      ...
      V3CipherSuites                  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
    }

In this example, EntireX Broker is configured to establish TCP/IP port 1958 and the Policy Agent gets instructions to make 1958 an SSL server port.

Therefore, section TTLSRule needs LocalPortRangeRef set to port 1958 and RemotePortRangeRef set to ports 1024-65535. The jobname of the Broker is needed, and the Direction of an SSL server port is Inbound.

The TTLSRule also refers to the following sections:

  • TTLSConnectionActionRef
    defines the HandshakeRole Server and refers to cipher parameters and advanced parameters for the connection

  • TTLSEnvironmentActionRef
    defines the HandshakeRole Server and refers to the Keyring parameter

  • TTLSGroupActionRef
    defines TTLSEnabled On

Advanced parameters for the connection define the sample CertificateLabel value ExxAppCert and the protocols to be used or not to be used.

The Keyring parameter specifies the RACF keyring using the following format: USER-ID / KEY-RING-NAME.

Setting up Entire Net-Work/Adabas SVC Transport

Start of instruction setTo set up EntireX Net-Work communication mechanism

  1. Ensure that all load libraries in the Broker kernel steplib are APF-authorized.

  2. Ensure that appropriate values are supplied in the Broker attribute file section DEFAULTS=NET, paying particular attention to the IUBL parameter - which specifies the maximum send/receive buffer length that can be sent between an application and Broker kernel within a single request - and NABS, which governs the total amount of storage available concurrently for all users communicating over this transport mechanism. See Adabas SVC/Entire Net-Work-specific Attributes.

  3. Ensure that communication with the EntireX Broker is possible by running the installation verification programs using transport type NET. See Sample Programs for Client (BCOC) and Server (BCOS) in the z/OS Installation documentation.

Starting and Stopping the Broker

Start of instruction setTo start the Broker

  1. Create a user ID for the started task or job where your Broker kernel is going to run.

  2. If you are using a started task, ensure that the user ID is defined in the list of user IDs for started tasks and that the sample Broker kernel JCL is modified appropriately to create a PROC.

  3. Start the Broker kernel either from the Broker kernel job (JCL) or started task (PROC).

Start of instruction setTo stop the Broker

  • Issue the operator command P <JOBNAME>

    Or:
    Execute the ETBCMD utility using the example syntax below:

    //ETBCMD  EXEC PGM=ETBCMD,
    // PARM=('/-bhost:port:TCP ',
    //       '-cSHUTDOWN -dBROKER -xuid -ypwd')

See Operator Commands for a full list and also Broker Command-line Utilities.

Dual Command Log Files

Command logging is a feature to assist in debugging Broker ACI applications. A command in this context represents one user request sent to the Broker and the related response of Broker.

Broker uses two command log files, enabling data to be written to one of the files while the other is being copied for archival purposes. Two file names must be specified for the dual command logs. At startup, Broker initializes both files and keeps the first open for printing command log data. Broker kernel switches to the other command log when the first file becomes full - or when the size of the open file reaches the value optionally specified by CMDLOG-FILE-SIZE (specified in KB).

Note:
It is always advisable to copy the contents of a full command log file before Broker fills the subsequent command log file. Otherwise, the information in the first file (full and closed) will be overwritten.

The file requirements are two equally sized, physical sequential files defined with a record length of 121 bytes, i.e.
DCB=(LRECL=121,RECFM=PS,BLKSIZE=nnnn). We recommend you allocate files with a single (primary) extent only. For example SPACE=(CYL,(30,0)). The minimum file size is approximately 3 cylinders of 3390 device.

Alternatively, the dual command log files can be allowed in USS HFS file system.

For more information, see Command Logging in EntireX.

Tracing EntireX Broker

This section covers the following topics:

Broker TRACE-LEVEL Attribute

The Broker TRACE-LEVEL attribute determines the level of tracing to be performed while Broker is running. The Broker has a master TRACE-LEVEL specified in the Broker section of the attribute file as well as several individual TRACE-LEVEL settings that are specified in the following sections of the attribute file.

Individual Settings Specified in Attribute File Section Note
Master trace level DEFAULTS=BROKER 1,2
Persistent store trace level DEFAULTS=ADABAS | CTREE | DIV 1
Conversion trace level DEFAULTS=SERVICE; Trace option of the service-specific broker attribute CONVERSION.  
Security trace level DEFAULTS=SECURITY 1
Transport trace level DEFAULTS=NET | TCP | SSL 1
Application Monitoring trace level DEFAULTS=APPLICATION-MONITORING  

Notes:

  1. For temporary changes to the master or individual TRACE-LEVEL without restarting the Broker, use the Broker command-line utility ETBCMD.
  2. For temporary changes to the master TRACE-LEVEL without restarting the broker, use operator command TRACE.

Attribute File Trace Setting

Trace Level Description
0 No tracing. Default value.
1 Traces incoming requests, outgoing replies, and resource usage.
2 All of Trace Level 1, plus all main routines executed.
3 All of Trace Level 2, plus all routines executed.
4 All of Trace Level 3, plus Broker ACI control block displays.

Note:
Trace levels 2 and above should be used only when requested by Software AG Support.

Deferred Tracing

It is not always convenient to run with TRACE-LEVEL defined, especially when higher trace levels are involved. Deferred tracing is triggered when a specific condition occurs, such as an ACI response code or a broker subtask abend. Such conditions cause the contents of the trace buffer to be written, showing trace information leading up the specified event. If the specified event does not occur, the Broker trace will contain only startup and shutdown information (equivalent to TRACE-LEVEL=0). Operating the trace in this mode requires the following additional attributes in the broker section of the attribute file. Values for TRBUFNUM and TRAP-ERROR are only examples.

Attribute Value Description
TRBUFNUM 3 Specifies the deferred trace buffer size = 3 * 64 K.
TRMODE WRAP Indicates trace is not written until an event occurs.
TRAP-ERROR 322 Assigns the event ACI response code 00780322 "PSI: UPDATE failed".

Dynamically Switching On or Off the EntireX Broker Trace

The following methods are available to switch on or off the EntireX Broker trace dynamically. You do not need to restart the broker for the changes to take effect.

  • ETBCMD
    Run command utility ETBCMD with option -c TRACE-ON or - c TRACE-OFF. See ETBCMD.

  • Operator Command
    Issue an operator command. See TRACE.

    See also Deferred Tracing.

Flushing Trace Data to a GDG Data Set

With broker-specific attributes TRMODE=WRAP and TRBUFNUM=n, Broker writes trace data to internal buffers instead of stderr (DD:SYSOUT). These buffers are used in round-robin mode and do not involve any I/O operation. If you need trace data for diagnostic purposes, use the operator command FLUSH to write the trace data from the internal buffers to a data set. A FLUSH command is performed automatically in case of error exceptions. The output data set is not readable for any other user when the broker is running. To avoid this problem, you can use a GDG (generation data group) data set as output data set. First you must allocate the GDG and define it to the broker. These preparatory steps are outlined below. The GDG name EXX.GDG is used in the examples.

Note:
GDG is supported for deferred tracing only.

Allocating a GDG

Define the GDG with IDCAMS. This definition is needed before working with the GDG data sets. You can use the following JCL to define a GDG. The LIMIT parameter is set to 16, but may contain other values according to your needs.
//IDCAMS   EXEC PGM=IDCAMS,REGION=4096K
//SYSPRINT DD   SYSOUT=* 
//SYSIN    DD   * 
  DEFINE GENERATIONDATAGROUP -
  (NAME(EXX.GDG) -
  NOEMPTY -
  SCRATCH -
  LIMIT(16))
/*

Defining the GDG to Broker

The GDG data set as target for trace data can be managed without changes to the Broker JCL. However, the DD statement for such a GDG data set has to be defined as a Broker attribute in order to propagate the file characteristics to the runtime library of the IBM Language Environment.

TRACE-DD = "DSNAME=EXX.GDG, 
            DCB=(BLKSIZE=1210,DSORG=PS,LRECL=121,RECFM=FB),
            DISP=(NEW,CATLG,CATLG), 
            SPACE=(CYL,(100,10)),
            STORCLAS=SMS"

See TRACE-DD under Broker-specific Broker Attributes.

Writing to the GDG Data Set

After successful broker initialization, a new data set of the GDG is allocated and opened. Based on the defined GDG name EXX.GDG in the sample JCL above, data set names EXX.GDG.G0001V00, EXX.GDG.G0002V00, EXX.GDG.G0003V00 and so on will be allocated and written.

Use operator command FLUSH to write all trace data from internal buffers to the GDG data set. The data set will be closed at the end of the FLUSH processing, and the next GDG data set is allocated and opened. During broker shutdown, the GDG data set is filled with all available trace data and closed.

Protecting a Broker against Denial-of-Service Attacks

An optional feature of EntireX Broker is available to protect a broker running with SECURITY=YES against denial-of-service attacks. An application that is running with invalid user credentials will get a security response code. However, if the process is doing this in a processing loop, the whole system could be affected. If PARTICIPANT-BLACKLIST is set to YES, EntireX Broker maintains a blacklist to handle such "attacks". If an application causes ten consecutive security class error codes within 30 seconds, the blacklist handler puts the participant on the blacklist. All subsequent requests from this participant are blocked until the BLACKLIST-PENALTY-TIME has elapsed.

Server Shutdown Use Case

Here is a scenario illustrating another use of this feature that is not security-related.

An RPC server is to be shut down immediately, using Broker Command and Information Services (CIS), and has no active request in the broker. The shutdown results in the LOGOFF of the server. The next request that the server receives will probably result in message 00020002 "User does not exist", which will cause the server to reinitialize itself. It was not possible to inform the server that shutdown was meant to be performed.

With the blacklist, this is now possible. As long as the blacklist is not switched off, when a server is shut down immediately using CIS and when there is no active request in the broker, a marker is set in the blacklist. When the next request is received, this marker results in message 00100050 "Shutdown IMMED required", which means that the server is always informed of the shutdown.

Setting the Time Zone for Broker

Broker obtains the time zone value by reading the environment variable TZ. If not specified, Broker retrieves the assignment of TZ from the configuration file /etc/profile.

Check your /etc/profile for an appropriate setting of environment variable TZ. The TZ value should reflect the appropriate value for your location.

Remember that the new Daylight Saving Time rule according to the Energy Policy Act in the U.S. takes effect in 2007. If you live in an area affected by the new rule, you may append it to the TZ environment variable.

For example, TZ=EST5EDT,M4.1.0,M10.5.0 is no longer valid. TZ=EST5EDT,M3.2.0,M11.1.0 must be used instead.

If you don't want to change your /etc/profile, you may configure Broker's startup JCL to define environment variable TZ. Modify the EXEC statement thus:

//BROKER   EXEC PGM=ETBNUC,REGION=0M,TIME=1440,
//         PARM='ENVAR(''TZ=EST5EDT,M3.2.0,M11.1.0'')/'

The value of TZ should reflect the appropriate value for your location. Put the time zone value in quotation marks and parentheses, as outlined above. The rule for daylight saving time changes can be appended after the time zone value.

Achieving FIPS Compliance

The z/OS operating system can be configured to run EntireX Broker FIPS 140-2 compliant. This requires changes in the following system components:

  • AT-TLS (Application Transparent Transport Layer Security)

  • System SSL

  • ICSF (z/OS Integrated Cryptographic Service Facility)

The TCP/IP port of Broker must run under the control of AT-TLS, and all applications using Broker need the Secure Socket Layer protocol (TLS/SSL) as transport. You do not need to change the attribute file of Broker.

The IBM support pages provide the document Setting up AT-TLS for FIPS 140 mode.pdf. It contains an overview of necessary system changes and describes the steps in detail.