Diagnosing infinite loops in the correlator
A correlator live lock occurs when events are recursively routed without a termination mechanism. The following example shows this in its simplest form:
on all Foo() {
route Foo();
}
More complex forms might recurse after a connected chain of several events being routed between different monitors.
There are no limits on how many routed events can be queued. Consequently, depending on the nature of the bug, the correlator might run out of memory. Note that an overloaded correlator would show similar symptoms, but can be distinguished by the fact that work is still being done (events are being sent out from the correlator).
When the correlator is in an infinite loop, it quickly uses an entire CPU and if there are events being routed as part of the loop then the correlator will run out of memory. Use the following correlator diagnostics to diagnose an infinite loop:
rq —sum of the number of routed events on the input queues of all contexts. When the correlator is in an infinite loop, this will always be 1 or it will always be increasing. It depends on the application.
iq — sum of the number of entries on the input queues of all contexts. When the correlator is in an infinite loop, this number is continuously increasing.
tx — number of transmitted events. This number is static when the correlator is in an infinite loop.
To identify an infinite loop in a particular context, run engine_inspect -x a few times. This lists each context along with the number of events on its input queue. See if there are contexts that have input queues that are getting bigger and bigger.