Thread Pool Monitoring

In addition to the thread dump generation provided by the Java Service Wrapper, you can configure the Universal Messaging realm servers to monitor the thread pool for slow-running threads and generate thread dumps when certain events occur. The thread dumps and messages generated from the user-defined monitoring of the thread pool are logged into Software AG_directory \UniversalMessaging\server\InstanceName\data\nirvana.log.

The thread pool monitoring generates thread dumps for stalled or slow-moving tasks, and reports reduced thread availability. Stalled tasks are tasks that run longer than the specified time. Slow-moving tasks are tasks that run slower than the timeout for the task execution. You can also set a threshold for pending tasks to monitor thread availability. You can then use the thread dump entries in the log to troubleshoot the task execution.

You configure the thread-pool monitoring properties for a realm in the Thread Pool Config group on the Config tab in the Enterprise Manager.

The time in milliseconds before reporting a stalled task. The system writes the information at the WARNING log level and generates a thread dump. When you change this configuration, the thread pool monitor interval is updated to monitor at the same time interval as the value you specify for this property. Valid values range from 10000 to 60000. Default is 60000.

The time in milliseconds before reporting a slow-running task. The server logs the information at the WARNING log level and generates a thread dump. Valid values range from 1000 to 30000. Default is 5000.

The threshold at which the server starts to warn about the number of pending tasks. When the number of pending tasks is below the threshold, but over 100, the server logs a WARNING message. When the number is above the threshold, the server logs an ERROR message. When the server does not find available threads, it logs a message that the thread pool is exhausted. Valid values range from 100 to 100000. Default is 1000.

Whether to generate a thread dump when the server reports a slow task. Valid values are true - generate a thread dump, or false - do not generate a thread dump. Default is false.

The interval in milliseconds at which a thread dump is generated when the system reports slow or stalled tasks, or when the number of pending tasks exceeds the value of PendingTaskWarningThreshold. The thread dump interval applies across all thread pools in the JVM instance. Valid values range from 1000 to 600000. Default is 60000.

In the following example, the slow-moving task warning timeout is set to 1000ms and the server is configured to generate a thread dump for a slow task. The task completed in 1060ms and the server reports that the task execution time exceeds 1000ms and generates a thread dump. The following entries will show in the log:

[Wed Feb 17 07:39:32.790 IST 2021] [ThreadPoolTest-Slow:9] ThreadPool:
<ThreadPool-SlowTasksTest> Slow moving task detected. ThreadPool-SlowTasksTest:9 has been active
for over 1060(ms) running task class
com.pcbsys.foundation.threads.fThreadPoolTaskReportTest$TestTask, Idle Threads 7, Allocated
Threads 10, Queued Tasks 0, Task Executed 10
[Wed Feb 17 07:39:32.790 IST 2021] [ThreadPoolTest-Slow:0] ThreadPool: <ThreadPool-SlowTasksTest>
Slow moving task detected. ThreadPool-SlowTasksTest:0 has been active for over 1060(ms) running
task class com.pcbsys.foundation.threads.fThreadPoolTaskReportTest$TestTask, Idle Threads 7,
Allocated Threads 10, Queued Tasks 0, Task Executed 10

[Wed Feb 17 07:39:41.000 IST 2021] [ThreadPoolTest-Slow:7] ThreadPool:
<TThreadPool-SlowTasksTest> Slow moving task detected. ThreadPool-SlowTasksTest:7 has been active
for over 1020(ms) running task class
com.pcbsys.foundation.threads.fThreadPoolTaskReportTest$TestTask, Idle Threads 7, Allocated
Threads 10, Queued Tasks 0, Task Executed 10
[Wed Feb 17 07:39:41.000 IST 2021] [ThreadPoolTest-Slow:7] Producing thread dump. Reason : Slow
moving task detected on thread pool: ThreadPool-SlowTasksTest

In the following example, the threshold for pending tasks is set to 200. Because the number of pending tasks in the thread pool is 209, the server logs an error message and generates a thread dump. The following entries will show in the log:

[Wed Feb 17 07:39:39.949 IST 2021] [Time-limited test] ThreadPool:
<ThreadPoolTest-Pending> Pending tasks are above the threshold 200 pending tasks 209, Idle Threads
0, Allocated Threads 1, Queued Tasks 209, Task Executed 210
[Wed Feb 17 07:39:39.949 IST 2021] [Time-limited test] Producing thread dump. Reason : Pending
tasks are above the threshold: 200 pending tasks: 209

In the thread pool configuration, check whether you have allocated enough threads. If the logs report a large number of queued tasks, allocate more threads to the pool.