List of correlator status statistics
This topic gives a detailed list of the status values that can be monitored for a correlator. The descriptions below show where the status values are used. The status is available through the following mechanisms:
Java API: The name of the method in the
EngineStatus Java API (and also the equivalent, where available, in the C++, .NET and EPL APIs). See the
com.apama.engine.EngineStatus interface in the
API Reference for Java (Javadoc).
Note: In the C++ API, the status statistics names are defined as constants inside the
client_status_names.hpp file. See also the
API Reference for C++ (Doxygen).
Prometheus metric name: The name used to expose internal correlator statistics to the Prometheus monitoring system. See also
Monitoring with Prometheus.
The descriptions below also indicate the typical trend. This can be one of the following:
Steady: After any start-up phase, this number would typically be steady. It may increase as bursts of events come in, or if there is a change in the size of the application (for example, the number of items the application is tracking). Typically, if these numbers are continually trending upwards when there is no more being asked of the application, that indicates an application leak of monitor instances, listeners or objects. This will eventually lead to an out of memory condition.
Increasing: This may be increasing in normal usage. Depending on deployment, some statistics may not be increasing, though if they normally are and have stopped increasing, this may indicate that something is preventing events being delivered or processed correctly.
Low: This number is typically 0 or near 0. If this number increases, this typically indicates that the correlator is not keeping up with processing events. For queues, it is normal that during bursts of activity, these may be non-zero for some time. Steadily increasing queue sizes can be a sign of back-pressure due to a slow receiver, or the system is not keeping up and may eventually block senders due to not processing the events at the rate they arrive.
Varies: Will typically vary. 0 may indicate a problem with events being delivered.
None: Typically, all contexts and receivers should be keeping up, so none are reported as slow (in which case, the empty string will be returned from the API).
The term "receiver" which is used in the descriptions below refers to any of the following:
EPL, Java or C++ plug-ins using the
Correlator.subscribe method.
Connectivity plug-ins for "towards" transport events.
JMS connections sending events out of the correlator.
Client library connections, including other correlators that have been connected with the
engine_connect,
iaf or
engine_receive tools.
Time since the correlator was started
The time in milliseconds since the correlator was started.
Typical trend: increasing.
REST API:
uptimeJava API:
getUptimeLog field:
not applicablePrometheus metric name:
sag_apama_correlator_uptime_secondsDisplay name:
Uptime (ms)Number of contexts
The number of contexts in the correlator, including the main context.
Typical trend: steady.
REST API:
numContextsJava API:
getNumContextsLog field:
nctx=nPrometheus metric name:
sag_apama_correlator_contexts_totalDisplay name:
Number of contextsNumber of monitors
The number of EPL monitor definitions injected into the correlator. This number changes on injections, deletions or if the last instance of a monitor terminates.
Typical trend: steady.
REST API:
numMonitorsJava API:
getNumMonitorsLog field:
not applicablePrometheus metric name:
sag_apama_correlator_monitors_totalDisplay name:
Number of monitorsNumber of monitor instances
The number of monitor instances, also known as sub-monitors.
Typical trend: steady.
REST API:
numProcessesJava API:
getNumProcessesLog field:
sm=nPrometheus metric name:
sag_apama_correlator_monitor_instances_totalDisplay name:
Number of sub-monitorsNumber of Java applications and Java EPL plug-ins
The number of Java applications and Java EPL plug-ins loaded in the correlator. This number changes on injections and deletions.
Typical trend: steady.
REST API:
numJavaApplicationsJava API:
getNumJavaApplicationsLog field:
not applicablePrometheus metric name:
sag_apama_correlator_java_applications_totalDisplay name:
Number of Java applicationsNumber of listeners
The number of listeners in all contexts. This includes on statements and active stream source templates.
Typical trend: steady.
REST API:
numListenersJava API:
getNumListenersLog field:
ls=nPrometheus metric name:
sag_apama_correlator_listeners_totalDisplay name:
Number of listenersNumber of sub-listeners
The number of sub-event-listeners that are active across all contexts. Stream source templates will have one sub-event-listener. An
on statement can have multiple sub-event-listeners. See also
Evaluating event listeners for all A-events followed by
B-events.
Typical trend: steady.
REST API:
numSubListenersJava API:
getNumSubListenersLog field:
not applicablePrometheus metric name:
sag_apama_correlator_sub_listeners_totalDisplay name:
Number of sub-listenersNumber of event types
The number of event types defined within the correlator. This number changes on injections and deletions.
Typical trend: steady.
REST API:
numEventTypesJava API:
getNumEventTypesLog field:
not applicablePrometheus metric name:
sag_apama_correlator_event_types_totalDisplay name:
Number of event typesNumber of executors on input queues
The number of executors on the input queues of all contexts. As well as events, this can include clock ticks, spawns, injections and other operations. A context in an infinite loop will grow by 10 per second due to clock ticks. Every context has an input queue, which by default is a maximum of 20,000 entries.
Typical trend: low.
REST API:
numQueuedInputJava API:
getNumQueuedInputLog field:
iq=nPrometheus metric name:
sag_apama_correlator_queued_input_totalDisplay name:
Events on input queueNumber of received events
The number of events that the correlator has received from external sources since the correlator started. This includes connectivity plug-ins, correlator-integrated JMS, engine_send, other correlators connected with engine_connect, dashboard servers, the IAF, and events that are not parsed correctly. This number excludes events sent within the correlator from EPL monitors or EPL plug-ins.
Typical trend: increasing.
REST API:
numReceivedJava API:
getNumReceivedLog field:
rx=nPrometheus metric name:
sag_apama_correlator_input_totalDisplay name:
Events receivedNumber of processed events
The number of events processed by the correlator in all contexts. This includes external events and events routed to contexts by monitors. An event is considered to have been processed when all listeners and streams that were waiting for it have been triggered, or when it has been determined that there are no listeners for the event.
Typical trend: increasing.
REST API:
numProcessedJava API:
getNumProcessedLog field:
not applicablePrometheus metric name:
sag_apama_correlator_processed_totalDisplay name:
Events processedSum of events on route queues
The sum of routed events on the route queues of all contexts.
Typical trend: low.
REST API:
numQueuedFastTrackJava API:
getNumQueuedFastTrackLog field:
rq=nPrometheus metric name:
sag_apama_correlator_queued_route_totalDisplay name:
Events on internal queueNumber of routed events
The number of events that have been routed across all contexts since the correlator was started.
Typical trend: increasing.
REST API:
numFastTrackedJava API:
getNumFastTrackedLog field:
rt=nPrometheus metric name:
sag_apama_correlator_route_totalDisplay name:
Events routed internallyNumber of external consumers/receivers
The number of external consumers/receivers connected to receive emitted events. This includes connectivity plug-ins, correlator-integrated JMS, engine_receive, or correlators connected using engine_connect.
Typical trend: steady.
REST API:
numConsumersJava API:
getNumConsumersLog field:
nc=nPrometheus metric name:
sag_apama_correlator_consumers_totalDisplay name:
Number of consumersNumber of events on output queues
The number of events waiting on output queues to be dispatched to any connected external consumers/receivers.
Typical trend: low.
REST API:
numOutEventsQueuedJava API:
getNumOutEventsQueuedLog field:
oq=nPrometheus metric name:
sag_apama_correlator_queued_output_totalDisplay name:
Events on output queueNumber of events created for sending to external channels
The number of events that have been sent (see
The send . . . to statement) or emitted (see
The emit statement) to channels which have at least one external consumer/receiver subscribed (see also
Number of external consumers/receivers). This excludes events sent to channels with no external consumers/receivers. This counts each event once, even if delivered to multiple external consumers/receivers.
Typical trend: increasing.
REST API:
numEmitsJava API:
getNumOutEventsCreatedLog field:
not applicablePrometheus metric name:
sag_apama_correlator_created_output_totalDisplay name:
Output events createdNumber of events delivered to external consumers/receivers
The number of events that have been delivered to external consumers/receivers. This counts for each external consumer/receiver an event is sent to. It counts the number of deliveries of events.
Note: This status indicator counts every event that was delivered, whereas the previous status indicator counts every event that was sent. For example, sending one event to a channel with two external consumers/receivers would be counted as one event sent (numEmits), but two events delivered (numOutEventsSent).
Typical trend: increasing.
REST API:
numOutEventsSentJava API:
getNumOutEventsSentLog field:
tx=nPrometheus metric name:
sag_apama_correlator_output_totalDisplay name:
Output events sentNumber of events on input queues of all public contexts
The number of events on the input queues of all public contexts. See also About context properties for information on the receiveInput flag. Typical trend: low.
REST API:
numInputQueuedInputJava API:
getNumInputQueuedInputLog field:
icq=nPrometheus metric name:
sag_apama_correlator_queued_input_public_totalDisplay name:
Events on input context queuesName of slowest context
The name of the slowest context. This may or may not be a public context.
Typical trend: none.
REST API:
mostBackedUpInputContextJava API:
getMostBackedUpInputLog field:
lcn=namePrometheus metric name: The name of the slowest context is given as a Prometheus label on the Prometheus metric
sag_apama_correlator_slowest_input_queue_size_totalDisplay name:
Slowest context nameNumber of events on queue for slowest context
The number of events on the slowest context's queue, as identified by the name of the slowest context.
Typical trend: low.
REST API:
mostBackedUpICQueueSizeJava API:
getMostBackedUpQueueSizeLog field:
lcq=nPrometheus metric name:
sag_apama_correlator_slowest_input_queue_size_totalDisplay name:
Slowest context queue sizeTime difference in seconds for slowest context
For the context identified by the slowest context name, this is the time difference in seconds between its current logical time and the most recent time tick added to its input queue.
Typical trend: low.
REST API:
mostBackedUpICLatencyJava API:
getMostBackedUpICLatencyLog field:
lct=secondsPrometheus metric name:
sag_apama_correlator_slowest_input_queue_latency_secondsDisplay name:
not applicableName of slowest consumer/receiver of events
The name of the consumer/receiver with the largest number of incoming events waiting to be processed. This is the slowest non-context consumer/receiver of events, which can be an external receiver or an EPL plug-in.
Typical trend: none.
REST API:
slowestReceiverJava API:
getSlowestReceiverLog field:
srn=namePrometheus metric name: The name of the slowest consumer/receiver of events is given as a Prometheus label on the Prometheus metric
sag_apama_correlator_slowest_output_queue_size_totalDisplay name:
Slowest receiver nameNumber of events on queue for slowest consumer/receiver
The number of events on the slowest consumer's/receiver's queue, as identified by the name of the slowest consumer/receiver.
Typical trend: low.
REST API:
slowestReceiverQueueSizeJava API:
getSlowestReceiverQueueSizeLog field:
srq=nPrometheus metric name:
sag_apama_correlator_slowest_output_queue_size_totalDisplay name:
Slowest receiver queue sizeNumber of events per second
The number of events per second currently being processed by the correlator across all contexts. This value is computed with every status refresh and is only an approximation.
Typical trend: varies.
REST API:
not applicableJava API:
not applicableLog field:
not applicablePrometheus metric name:
not applicableDisplay name:
Event rate over last intervalNumber of enqueued events
The number of events queued from the enqueue statement (not the enqueue...to statement). See also The enqueue statement. Typical trend: low.
REST API:
enqueueQueueSizeJava API:
not applicableLog field:
not applicablePrometheus metric name:
not applicableDisplay name:
not applicableVirtual memory
Virtual memory. For the REST API, the value is in megabytes. For the log field, the value is in kilobytes. For Prometheus, the value is in bytes.
Typical trend: steady.
REST API:
virtualMemoryMBJava API:
not applicableLog field:
vm=kBPrometheus metric name:
sag_apama_correlator_virtual_memory_bytesDisplay name:
not applicablePhysical memory
Physical memory. For the REST API, the value is in megabytes. For the log field, the value is in kilobytes. For Prometheus, the value is in bytes.
Typical trend: steady.
REST API:
physicalMemoryMBJava API:
not applicableLog field:
pm=kBPrometheus metric name:
sag_apama_correlator_physical_memory_bytesDisplay name:
not applicableNumber of contexts on run queue
The number of contexts on the run queue. These are the contexts that have work to do but are not currently running.
Typical trend: low.
REST API:
not applicableJava API:
not applicableLog field:
runq=nPrometheus metric name:
not applicableDisplay name:
not applicableNumber of pages read from swap space
The number of pages per second that are being read from swap space. If this is greater than zero, it may indicate that the machine is under-provisioned, which can lead to reduced performance, connection timeouts and other problems. Consider adding more memory, reducing the number of other processes running on the machine, or partitioning your Apama application across multiple machines.
Typical trend: low.
REST API:
swapPagesReadJava API:
not applicableLog field:
si=nPrometheus metric name:
sag_apama_correlator_swap_pages_read_hertzDisplay name:
not applicableNumber of pages written to swap space
The number of pages per second that are being written to swap space. If this is greater than zero, it may indicate that the machine is under-provisioned, which can lead to reduced performance, connection timeouts and other problems. Consider adding more memory, reducing the number of other processes running on the machine, or partitioning your Apama application across multiple machines.
Typical trend: low.
REST API:
swapPagesWriteJava API:
not applicableLog field:
so=nPrometheus metric name:
sag_apama_correlator_swap_pages_write_hertzDisplay name:
not applicableNumber of snapshots
The number of persistence snapshots taken since the correlator started. These statistics will only exist if the correlator is running in persistent mode.
Typical trend: increasing.
REST API:
persistenceNumSnapshotsJava API:
not applicableLog field:
numSnapshots=nPrometheus metric name:
sag_apama_correlator_persistence_snapshots_totalDisplay name:
not applicableTimestamp of last snapshot
The UNIX timestamp of the last completed snapshot. These statistics will only exist if the correlator is running in persistent mode.
Typical trend: increasing.
REST API:
persistenceLastSnapshotTimeJava API:
not applicableLog field:
lastSnapshotTime=timestampPrometheus metric name:
sag_apama_correlator_persistence_last_snapshot_timestampDisplay name:
not applicableTime waiting for a snapshot (EWMA)
An exponentially weighted moving average (EWMA) of the time in milliseconds taken to wait for a snapshot. These statistics will only exist if the correlator is running in persistent mode.
Typical trend: varies.
REST API:
persistenceSnapshotWaitTimeEwmaMillisJava API:
not applicableLog field:
snapshotWaitTimeEwmaMillis=millisecondsPrometheus metric name:
sag_apama_correlator_persistence_snapshot_wait_ewma_secondsDisplay name:
not applicableTime to commit to database (EWMA)
An exponentially weighted moving average (EWMA) of the time in milliseconds taken to commit to the database. These statistics will only exist if the correlator is running in persistent mode.
Typical trend: varies.
REST API:
persistenceCommitTimeEwmaMillisJava API:
not applicableLog field:
commitTimeEwmaMillis=millisecondsPrometheus metric name:
sag_apama_correlator_persistence_commit_time_ewma_secondsDisplay name:
not applicableNumber of changed rows (EWMA)
An exponentially weighted moving average (EWMA) of the number of rows changed per snapshot. These statistics will only exist if the correlator is running in persistent mode.
Typical trend: varies.
REST API:
persistenceLastSnapshotRowsChangedEwmaJava API:
not applicableLog field:
lastSnapshotRowsChangedEwma=nPrometheus metric name:
sag_apama_correlator_persistence_snapshot_rows_changed_ewma_totalDisplay name:
not applicableTotal heap memory used by the JVM
The total heap memory used by the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.
Typical trend: steady.
REST API:
jvmMemoryHeapUsedMBJava API:
not applicableLog field:
not applicablePrometheus metric name:
sag_apama_correlator_jvm_heap_used_bytesDisplay name:
not applicableTotal free heap memory in the JVM
The total heap memory that is free in the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.
Typical trend: steady.
REST API:
jvmMemoryHeapFreeMBJava API:
not applicableLog field:
not applicablePrometheus metric name:
sag_apama_correlator_jvm_heap_free_bytesDisplay name:
not applicableTotal non-heap memory used by the JVM
The total non-heap memory used by the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.
Typical trend: steady.
REST API:
jvmMemoryNonHeapUsedMBJava API:
not applicableLog field:
not applicablePrometheus metric name:
sag_apama_correlator_jvm_non_heap_used_bytesDisplay name:
not applicableTotal memory used by all buffer pools in the JVM
The sum of memory used by all buffer pools in the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.
Typical trend: steady.
REST API:
jvmMemoryBufferPoolUsedMBJava API:
not applicableLog field:
not applicablePrometheus metric name:
sag_apama_correlator_jvm_buffer_pool_used_bytesDisplay name:
not applicableTotal memory used by the JVM
The sum of all memory used by the Java virtual machine (JVM) which is embedded in the correlator (that is, the used heap memory, the used non-heap memory, and the used buffer pool memory). For the REST API and the log field, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API and the log field will return 0 (zero) as the value, and Prometheus will not have this metric.
Typical trend: steady.
REST API:
jvmMemoryAllUsedMBJava API:
not applicableLog field:
jvm=nPrometheus metric name:
sag_apama_correlator_jvm_memory_all_bytesDisplay name:
not applicable