Apama 10.3 | Apama Documentation | Introduction to Apama | Apama Architecture | How the correlator works
How the correlator works
The following figure shows the inner details of a running correlator. After the figure, there is a detailed discussion of how the correlator works.
Monitors and queries identify event patterns of interest and the responses to take if those patterns are detected. You can use EPL to write monitors and queries directly. You can use JMon to write monitors directly. Apama uses the Software AG Designer development environment for writing source code for monitors and queries, and provides a graphical editor for defining queries (Query Designer). When you use Query Designer, the query is translated into a monitor that the correlator can execute.
The correlator does not just execute loaded monitors and queries in a sequential manner, as if they were traditional imperative programs. Instead, the correlator loads its internal components (the hypertree and the temporal sequencer) with the monitoring specifications of the monitors and queries. The in-built virtual machines execute only the sequential analytic or action parts of the monitors and queries.
The correlator contains the following components:
*HyperTree multi-dimensional event matcher
The event matcher contains data structures and algorithms designed for high performance, multi-dimensional, event filtering. The correlator loads the event matcher with event templates. An event template identifies the event you are interested in. Logically, an event template is a multi-dimensional search. For example, a template for a stock market event might have values such as the following:
*Instrument: IBM
*Bid Price: 93.0 <- -> 94.5
*Offer Price: *
*Bid Volume: >10000
*Offer Volume: *
This event template expresses a multi-dimensional search over stock market events. The template will match any event about stock IBM, which has a bid price between 93.0 and 94.5 and a bid volume greater than 100000. The offer price and volume are irrelevant to this search and so wildcards are used.
This kind of multi-dimensional, multi-type, ranged searching is what the event matcher was specifically designed for. In checking whether an incoming event matches any of the registered event templates, the event matcher exhibits logarithmic performance. This means that vast numbers of event templates can be queried against, with the minimum possible performance tail-off.
An event template is the basic unit of monitoring. A simple monitor might have one or a few event templates. A more complex monitor might have many. A monitor needs to load event templates only when events that match the specification are relevant to the monitor: in a multi-stage monitor, a monitor can insert and remove several event templates as the monitoring requirements change.
*Temporal and stream sequencer
The temporal and stream sequencer builds upon the single event matching capabilities of the event matcher to provide multiple temporal event and stream correlations. With EPL or JMon, you can declare a temporal sequence such as “tell me when any news article event is followed within 5 minutes by a 5% fall in the price of the stock the news article was about”. This is a temporal sequence, with a temporal constraint. The sequence is a news article event, followed by the next stock price event, and then another stock price event with a price 5% less than the previous price event. The temporal constraint is that the last event occurs within 5 minutes of the first event.
The sequencer manages this temporal monitoring process, using the event matcher to monitor for appropriate event templates. This capability saves the programmer from having to encode such complex temporal logic through less intuitive imperative logic.
The correlator provides the capability for monitors to be injected as either EPL or Java bytecode. The number of monitors that can be loaded into a single correlator are only limited by memory size. When loaded, a monitor configures the hypertree and temporal sequencer with event templates for monitoring. The correlator stores the monitor internally and executes actions in the appropriate virtual machine in response to event detection.
Each monitor instance has its own address space within the correlator for storage of variables and other state. Monitor temporary storage size is limited only by the memory size of the host machine.
Queries provide a higher level, more declarative mechanism for detecting event patterns. Unlike monitors, they also support automatically storing their state in a distributed cache such as Terracotta DB's TCStore. TCStore allows queries to transparently and elastically scale out with the support of a JMS message bus such as Software AG's Universal Messaging. The storage size is thus limited by the distributed cache, which can exceed the memory size of a single host machine.
*Event input queue
External interfaces, such as adapters and connectivity plug-in chains, send events into the correlator. To start the monitoring process, the correlator injects each event, in the order in which it arrives, into the hypertree. Any matches filter through the temporal sequencer and invoke required actions in the virtual machines. Some actions might cause events to be queued for output. During peak event input flow, events might wait on an input queue for an extremely brief moment.
*EPL virtual machine
In response to detected event patterns of interest, the EPL virtual machine executes EPL. The fact that the correlator behaves this way, rather than continuously executing imperative code, is another reason for its high performance. Also, you can implement parallel processing in your applications so that the correlator can concurrently execute code in multiple monitors.
*Java virtual machine
The Java virtual machine is a standard JVM that has been embedded in the correlator. Thus any standard Java code features are accessible from monitors. The Java virtual machine behaves exactly as the EPL virtual machine in that the detection of event patterns of interest invokes code fragments.
*Event output queue
Monitor actions can output events to be communicated to other monitors or to external systems. When a monitor routes an event, the event goes to the front of the input queue. This ensures that any monitors that are listening for that event immediately detect it. When a monitor generates an event for an external receiver the event goes to an output queue for transmission to the appropriate registered party.
When you use the correlator in conjunction with connectivity plug-ins or the IAF, then an output event might represent an action on an external service. The connectivity plug-in or IAF transforms the output event into an invocation of the external service. An example is an event that places an order into the order book of a Stock Exchange.
*EPL plug-ins
It is possible to extend the capabilities of the correlator through an EPL plug-in. An EPL plug-in is an externally-linked software module that registers with the correlator through the correlator's extension API. EPL plug-ins are useful when programming libraries of useful real-time functions have been built up. These functions can be made available as objects that can be invoked by EPL actions.
Apama provides a number of standard EPL plug-ins:
*The MemoryStore plug-in lets monitors share in-memory data.
*The TimeFormat plug-in helps you format dates and times.
*State persistence
When the correlator shuts down the default behavior is that all state is lost. When you restart the correlator no state from the previous time the correlator was running is available. You can change this default behavior by using correlator persistence. Correlator persistence means that the correlator automatically periodically takes a snapshot of its current state and saves it on disk. When you shut down and restart that correlator, the correlator restores the most recent saved state.
To enable persistence, you indicate in your EPL code which monitors you want to be persistent. Optionally, you can write actions that the correlator executes as part of the recovery process. When code is injected for a persistence application, the correlator that the code is injected into must have been started with a persistence option. Persistent monitors must be written in EPL. State in JMon monitors cannot be persistent. State in chunks, with a few exceptions, also cannot be persistent.
You program the correlator by injecting monitors that you write in EPL or JMon, or by injecting queries.
When events are sent to the correlator, the correlator processes events by comparing the events to what listeners are active in the correlator. Each external event matches zero or more listeners. The correlator executes a matching event's associated listeners in a rigid order. The correlator completes the processing related to a particular event before it examines the next event. If the processing of an event generates another event that is routed to the correlator, the correlator processes all routed events before moving on to the next event in its queue. If a listener action block does not route events, the next external event is considered.

Copyright © 2013-2018 | Software AG, Darmstadt, Germany and/or Software AG USA, Inc., Reston, VA, USA, and/or its subsidiaries and/or its affiliates and/or their licensors.