Monitoring zIIP Usage

The purpose of Entire Net-Work for zIIP is to reduce the Entire Net-Work CPU consumption on general processors (GP) by offloading part of the processing to System z Integrated Information Processors (zIIP). Entire Net-Work for zIIP offers statistics about how much CPU time it has consumed on GPs and zIIPs and for which reasons it has executed on GPs. This document provides information on how to view and understand these zIIP-related statistics.

zIIP-Related Statistics

Entire Net-Work for zIIP shows zIIP-related statistics

See Understanding the zIIP-Related Statistics below for information on how to interpret these statistics.

Understanding the zIIP-Related Statistics

An Entire Net-Work nucleus started with ADARUN parameter ZIIP=YES prints the zIIP-related statistics during termination at the end of its session statistics. They show the following information:

Statistic Description
Total CPU time The total CPU time (GP plus zIIP) consumed by any dispatchable unit (TCB or SRB) running in the Entire Net-Work address space, comprising the non-enclave GP times and all enclave GP times and zIIP times
Non-enclave GP times The total CPU time (on general processors) consumed by any TCB in the Entire Net-Work address space that was not a member of a Workload Manager (WLM) enclave
All enclave GP times The total CPU time on general processors consumed by any dispatchable unit (TCB or SRB) in the Entire Net-Work address space that was a member of a WLM enclave
All enclave zIIP times The total CPU time on zIIPs consumed by any SRB in the Entire Net-Work address space that was a member of a WLM enclave, normalized to GP speed
All enclave zIIP times (%) The percentage of the total CPU time that the Entire Net-Work address space consumed on zIIPs, calculated as: All enclave zIIP times / Total CPU time * 100
Total enclave CPU time The total CPU time consumed by the WLM enclave created by Entire Net-Work for its entire session, comprising the CPU times consumed on GPs and on zIIPs
Enclave GP time The CPU time of the WLM enclave created by Entire Net-Work that was consumed on general processors
Enclave zIIP time The CPU time of the WLM enclave created by Entire Net-Work that was consumed on zIIPs, normalized to GP speed
Enclave zIIP time (%) The percentage of the enclave CPU time that Entire Net-Work consumed on zIIPs, calculated as: Enclave zIIP time / Total enclave CPU time * 100
Eligible zIIP CPU time The CPU time that Entire Net-Work was eligible to execute on zIIPs, comprising the actual Enclave zIIP CPU time and the Eligible zIIP CPU time on GP
Enclave zIIP time The CPU time of the enclave that Entire Net-Work consumed on zIIPs (same as the ‘Enclave zIIP CPU time’ under ‘Total enclave CPU time’)
Eligible zIIP time on GP The CPU time of the enclave that Entire Net-Work was eligible to execute on zIIPs but instead consumed on GPs because no zIIP was available
Eligible zIIP time on GP (%) The percentage of the eligible zIIP CPU time that Entire Net-Work instead consumed on GPs, calculated as: Eligible zIIP CPU time on GP / Eligible zIIP CPU time * 100
Mode switches The number of times Entire Net-Work switched into SRB mode to become eligible for execution on a zIIP or switched into TCB mode to perform operations that were incompatible with SRB mode
Parallel requests The number of times Entire Net-Work requested that the TCB perform an operation in parallel to its own processing in SRB mode
No free element for request The number of times the SRB had to wait for a free request element until it could issue a parallel request to the TCB
Parallel requests per TCB pause The average number of parallel requests processed by the TCB until it had to pause and wait for more work, calculated as: Parallel requests / Pause TCB (below)
Extended statistics
The following extended statistics were introduced for internal reporting and may be changed or removed in future releases.
Pause SRB The number of times the SRB was waiting for work
Release SRB The number of times the SRB was released to continue processing
Pause TCB The number of times the TCB was waiting for work
Release TCB The number of times the TCB was released to continue processing
Pause for wait The number of times Entire Net-Work had no work to do (i.e., was waiting for I/Os, new commands, or other events)
Release from wait The number of times Entire Net-Work was released to continue processing after an event had occurred
SRB/TCB scheduling by type of work
The following statistics "by type of work" show why the processing mode (SRB or TCB mode) was switched or a parallel request was issued. They indicate the reasons for the "Mode Switches" and "Parallel requests" reported above. Only categories with nonzero counts are shown. A selection of typical categories follows:
Miscellaneous The number of other, infrequent operations that required execution in TCB mode, performed via switches to TCB mode and back to SRB mode
Operator commands The number of operator commands whose processing required execution in TCB mode, performed via switches to TCB mode and back to SRB mode
Sequential writes The number of writes to a sequential dataset (e.g., DDPRINT), in most cases issued via parallel requests to the TCB
Timer services The number of timer operations that required execution in TCB mode, performed via switches to TCB mode and back to SRB mode
User exit N The number of times user exit N was called. User exits (including hyperexits and collation descriptor exits) are always called in TCB mode. User exits that may be called frequently are invoked via parallel requests; others, via switches to TCB mode and back to SRB mode
SET ZIIP operator command The number of times operator command SET ZIP=YES/NO resulted in a mode switch
ADATMZ call The number of times ADATMZ was called in TCB mode, performed via switches to TCB mode and back to SRB mode
SAF call The number of times an SAF authorization call was made in TCB mode, performed via switches to TCB mode and back to SRB mode
User exit The number of times a Net-Work user exit was called in TCB mode, performed via switches to TCB mode and back to SRB mode
FCTC Open/Close The number of times a mode switch was performed to open or close the Driver or a Link
FCTC Wait The number of times a mode switch was performed to issue a WAIT. May happen during link open or disconnect.
FCTC Purge The number of times a mode switch was performed to make a PURGE call. May happen during link disconnect.
FCTC Calldisp 1 The number of times a mode switch was performed to issue CALLDISP. CALLDISP is used when the main task must be paused to allow the Disabled Interrupt Exit (DIE) to complete an operation. This number should be small compared to the number of commands sent and received.
FCTC Calldisp 2 The number of times a mode switch was performed to issue CALLDISP. CALLDISP is used when the main task must be paused to allow the Disabled Interrupt Exit (DIE) to complete an operation. This number should be small compared to the number of commands sent and received.
CTCA Open/Close The number of times a mode switch was performed to open or close the Driver or a Link
CTCA Wait The number of times a mode switch was performed to issue a WAIT. May happen during link open or disconnect.
CTCA Purge The number of times a mode switch was performed to make a PURGE call. May happen during link disconnect.
CTCA Calldisp 1 The number of times a mode switch was performed to issue CALLDISP. CALLDISP is used when the main task must be paused to allow the Disabled Interrupt Exit (DIE) to complete an operation. This number should be small compared to the number of commands sent and received.
CTCA Calldisp 2 The number of times a mode switch was performed to issue CALLDISP. CALLDISP is used when the main task must be paused to allow the Disabled Interrupt Exit (DIE) to complete an operation. This number should be small compared to the number of commands sent and received.
XCF Open The number of times a mode switch was performed to open the Driver or a Link
XCF Close The number of times a mode switch was performed to close the Driver or a Link
VTAM Open The number of times a mode switch was performed to open the Driver or a Link
VTAM Close The number of times a mode switch was performed to close the Driver or a Link
TCP Open The number of times a mode switch was performed to open the Driver or a Link
TCP Resolve The number of times GETHOSTBYNAME or GETHOSTBYADDR was called in TCB mode, performed via switches to TCB mode and back to SRB mode
… (Other types of work that must be performed in TCB mode are reported if they occurred in the Entire Net-Work session.)
Enclave GP service units The GP CPU service units accumulated by the enclave created by Entire Net-Work
Enclave zIIP service units The zIIP CPU service units accumulated by the enclave created by Entire Net-Work, normalized to GP speed
GPs The number of general processors (GP) managed by the operating system
zIIPs The number of System z Integrated Information Processors (zIIP) managed by the operating system
zIIP SMT threads The number of simultaneous multithreading (SMT) threads per zIIP core
zIIP normalization factor The factor by which zIIP CPU times have been multiplied by z/OS to be comparable with the CPU times of the GPs, if the GPs are throttled

Utilities running with ZIIP=YES print similar statistics in their DDPRINT output dataset.

Notes:

  1. The "All enclave zIIP times (%)" figure shows in a nutshell how much of the CPU consumption by Entire Net-Work was actually offloaded to zIIPs.
  2. Entire Net-Work creates a Workload Manager enclave for its main task TCB and its companion SRB. It reports the CPU times on zIIPs and general processors consumed in this enclave under “Total enclave CPU time”. Other tasks in the Entire Net-Work address space — such as system SRBs used for asynchronous event processing (I/O completion, cross-memory posts, XCF/XES exits used in Adabas Cluster Services, etc.) run outside this enclave. The CPU times for all enclaves, tasks and SRBs running in the Entire Net-Work address space are shown under “Total CPU time” at the top of these statistics.
  3. Take into account that the TCB/SRB mode switches and parallel requests generate overhead that is also attributed to the GP and zIIP CPU times of Entire Net-Work for zIIP. For a more accurate assessment how much CPU time Entire Net-Work for zIIP saves on GPs, run the same, representative test workload both with ZIIP=YES and ZIIP=NO and compare the GP CPU times consumed in both scenarios. The GP CPU time savings will typically depend on the type of workload processed by Entire Net-Work.
  4. If the "Eligible zIIP CPU time on GP (%)" is non-negligible, it suggests that the available zIIPs in the host system are over-allocated. If their free capacity was higher, Entire Net-Work for zIIP could offload more of its work to the zIIPs.
  5. The number of "Parallel requests per TCB pause" indicates the level of parallelism that Entire Net-Work for zIIP could utilize by stringing TCB-mode operations together. A number close to 1 indicates low parallelism; a greater number, higher parallelism and a greater reduction of overhead.
  6. The "Pause for wait" count, relative to the number of I/Os (reported in the zIIP-related statistics under "EXCPs") and Entire Net-Work commands (reported higher up in the session statistics), indicates the level of parallelism that Entire Net-Work could utilize to process multiple commands concurrently. The higher the ratio of commands plus I/Os over "pauses for wait", the more work Entire Net-Work was able to do without pause (such as waiting for I/O completion or for the arrival of a new command).
  7. The counts of mode switches and parallel requests depend on the workload processed by Entire Net-Work - in particular, the number of system service calls that require TCB mode, the number of user exit calls, and the inherent parallelism of the workload. Aside from changing these aspects, little can be done in the configuration of Entire Net-Work for zIIP to influence the interplay between the SRB and the TCB.
  8. If the ZIIP parameter is changed to NO during an Entire Net-Work session, requesting that Entire Net-Work continue to run in TCB mode and not use zIIPs anymore, the then following processing will be charged to the TCB and counted under "All enclave GP times" and "Enclave GP CPU time". The proportions of "All enclave zIIP times (%)" and "Enclave zIIP CPU time (%)" will decrease correspondingly. These percentages show how much of the Entire Net-Work workload was actually executed on zIIPs, not how much could have been executed on zIIPs under other circumstances.

Example zIIP-Related Statistics

The following example output illustrates the zIIP-related statistics in the Entire Net-Work session statistics: 

NET0200I: zIIP-related statistics for Net-Work address space
NET0200I: Total CPU time 0:00:38.884
NET0200I:  Non-enclave GP times 0:00:05.514
NET0200I:  All enclave GP times 0:00:01.558
NET0200I:  All enclave zIIP times 0:00:31.812
NET0200I:  All enclave zIIP times (%) 81.81
NET0200I: zIIP-related statistics for Net-Work enclave
NET0200I:  Total enclave CPU time 0:00:33.370
NET0200I:  Enclave GP time 0:00:01.558
NET0200I:  Enclave zIIP time 0:00:31.812
NET0200I:  Enclave zIIP time (%) 95.33
NET0200I:  Eligible zIIP CPU time 0:00:31.818
NET0200I:  Enclave zIIP time 0:00:31.812
NET0200I:  Eligible zIIP time on GP 0:00:00.006
NET0200I:  Eligible zIIP time on GP (%) 0.01
NET0200I: Mode switches 36,904
NET0200I: Parallel requests 36
NET0200I:  No free element for request 0
NET0200I:  Parallel requests per TCB pause 4.50
NET0200I: Extended statistics
NET0200I:  Pause SRB 31
NET0200I:  Release SRB 1
NET0200I:  Pause TCB 8
NET0200I:  Release TCB 38
NET0200I:  Pause for wait 0
NET0200I:  Release from wait 0
NET0200I: SRB/TCB scheduling by type of work
NET0200I:  Miscellaneous 2
NET0200I:  Operator commands 2
NET0200I:  Sequential writes 36
NET0200I:  Timer services 7,830
NET0200I:  User exit 8 72
NET0200I:  ADATMZ call 28,996
NET0200I: Enclave GP service units 184,795
NET0200I: Enclave zIIP service units 176,165
NET0200I: GPs 6
NET0200I: zIIPs 1
NET0200I:  zIIP SMT threads 2
NET0200I: zIIP normalization factor 10.97

See Understanding the zIIP-Related Statistics above for information and advice on how to interpret these statistics.