When running queries across multiple correlators in a cluster, as well as configuring all correlators to access the same distributed MemoryStore, Apama recommends that all events are delivered into the cluster using a JMS queue. By using a JMS queue, each correlator will pull events from the JMS queue unless it has a full input queue (that is, it is behind on processing events) or has stopped running (for example, shut down for maintenance or suffered a hardware failure). In either case, events will continue to be processed by other correlators in the cluster. Correlators can also be added to or removed from the cluster to scale the cluster capacity if desired. It is also possible to use per-correlator adapters for incoming events, but the adapters must co-ordinate so that every event is sent to only one correlator, and should one adapter/correlator pair fail, then other adapters process events that the failed node would have processed. Each event should only be delivered to one correlator, else multiple correlators will store the event in the shared cache, which can result in erroneous matches. Using JMS queues, this happens automatically, giving an elastic system that can be scaled and continues running in the face of failure.

To run queries across multiple correlators in a cluster:

When the cluster uses a JMS queue, each correlator pulls events from the queue. If the input queue of one correlator in the cluster becomes full and it cannot pull events from the JMS queue, the other correlators continue to do so and continue to process events. A correlator may stop pulling events because the correlator is behind on processing events or because it has stopped running, perhaps for maintenance or because of a hardware failure.

Using a JMS queue makes it easy to scale the cluster capacity by adding or removing correlators.

An alternative to using a JMS queue is to use an adapter for each correlator. For example, by having an IAF-based adapter connected to each correlator, it is possible to send messages to and from a query application without using JMS. A disadvantage of using per-correlator adapters is that the adapters must coordinate the following:

Use of a JMS queue automatically ensures that an event goes to only one correlator and that all received events are processed. The result is an elastic system that can be scaled and that continues to run even if a node fails.

Similar to using multiple contexts in a correlator, delivering events through JMS can result in events that occur close together in time being processed in an order that is different than the order in which they were created or sent to the JMS message bus.

Messages may be lost in the event of node failure, unless you have configured JMS for reliable message delivery (see also Handling node failure and failover).

Configure your JMS bus to have one or more queues, and configure a static JMS receiver connection. See Getting started with simple correlator-integrated messaging for JMS. You will also need to provide mapping for all event types that flow into the queries. See Mapping Apama events and JMS messages.

The queries runtime ensures that after all queries have been injected into the correlator and started, they automatically start to receive events from JMS queues. There is no need to explicitly call jms.onApplicationInitialized() as described in Using EPL to send and receive JMS messages.

For all applications that do not consist entirely of queries, for example, applications that contain additional EPL monitors or Java monitors, then it may be required to delay starting JMS until the application and queries are both ready to process events. The auto-starting of JMS behavior of queries can be controlled by sending a QueriesShouldNotAutoStartJMS() event to the main context. This event can be routed by an application's onload() method. If this is done, then a monitor in the main context should listen for a QueriesStarted() event and should wait until both the application and queries have started. The monitor can then call jms.onApplicationInitialized() directly. For example, the following monitor delays starting JMS until queries are started and a StartMyApp() event has been processed: