Deploying and Managing Apama Applications > Tuning Correlator Performance > Event correlator pipelining
Event correlator pipelining
To implement engine topologies comprising multiple event correlators requires a method of connecting correlators in pipelined configurations. This can be achieved in the following ways:
*Directly using the engine_connect tool, see Configuring pipelining with engine_connect.
*Indirectly using Software AG's Universal Messaging (UM) message bus. For complex deployments where parts of the application may be moved between Apama correlators, this is likely to be the best alternative. When using UM each correlator connects to the same UM realm.
See Using Universal Messaging in Apama Applications.
*Programmatically via the C++ client API, see Configuring pipelining through the client API.
*Using the Software AG Designer, click Connections on the Components tab of the Run Configurations dialog.
See Connecting correlators.
Configuring pipelining with engine_connect
The engine_connect tool allows direct connection of running correlator instances.
The tool is located in the bin folder of the Apama installation, as engine_connect.exe (on Windows) or engine_connect (on UNIX). Running the tool in the Apama Command Prompt (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.
Synopsis
Usage: engine_connect [ options ]
To connect a correlator, as an event receiver, to another correlator.
Where options include:
-h | --help This message
-sn | --sourcehost <host> Source engine on <host>
-sp | --sourceport <port> Source engine is listening on <port>
-tn | --targethost <host> Target engine on <host>
-tp | --targetport <port> Target engine is listening on <port>
-c | --channel <channel> Listen on output channel <channel>
-m | --mode <mode> Specify legacy or parallel
-x | --disconnect Destroy rather than create connections
-s | --qdisconnect Disconnect if slow (only takes effect on
first connection)
-f | --filename <file> Read config data from a file
-u | --utf8 Assume config file is in UTF8
-v | --verbose Be more verbose
-V | --version Print program version info
The target is connected as an event consumer to the source
Multiple -c options may be given
To read from stdin use -f -
Description
engine_connect connects a source correlator (the sender) to a target correlator (the receiver). The target correlator will receive events from the specified channel(s) of the source correlator. Source and target correlators must already be running.
Alternatively, if you specify the –f option, engine_connect reads connection information from the specified file and sets up each connection found therein (see Configuring pipelining through the client API for details of the file format). The engine_connect utility expects the specified file to be in the local character set. If the configuration file is in UTF-8, specify the -u option in addition to the -f option. If the filename provided to -f is ‘-’ then connection information is read from the standard input device (stdin) until end-of-file.
The connection between the source and target correlators is persistent. When one of the correlators stops running then when that correlator restarts it automatically reconnects with the other correlator.
The tool is silent by default unless an error occurs. For verbose progress information use the –v option.
Options
Option
Description
-h
Display usage information
-sn host
Name of the host on which the source (event sending) correlator is running. The default is localhost. However, you can use the default or specify localhost only when the source correlator and the target correlator are running on the same host. In all other situations, you must specify the public IP address or the name of the host. This ensures that the host of the target correlator can resolve the name/address of the source correlator host. Non-ASCII characters are not allowed in host names.
-sp port
Port on which the source (event sending) correlator is listening (default is 15903)
-tn host
Name of the host on which the target (event receiving) correlator is running. The default is localhost. However, you can use the default or specify localhost only when the source correlator and the target correlator are running on the same host. In all other situations, you must specify the public IP address or the name of the host. This ensures that the host of the source correlator can resolve the name/address of the target correlator host. Non-ASCII characters are not allowed in host names.
-tp port
Port on which the target (event receiving) correlator is listening (default is 15903)
-c channel
Named channel on which to send/receive events. You can specify the -c option multiple times to send/receive events on multiple channels. You must specify the -c option at least once for each sender/target pair. Until you do, no events emitted by the sender correlator are received by the target correlator. An event is discarded if it is sent on a channel for which you did not specify the -c option.
-m mode
Indicates whether there is one connection (-m legacy) between the sender and target correlators or one connection for each specified channel (-m parallel).
The default behavior is that there is one connection between the sender and target correlators. The utility uses the same connection for every channel. Events sent on any channel are delivered to the default channel in the target correlator and all events are delivered in order. You can specify default behavior by specifying -m legacy or --mode legacy.
To create a connection for each specified channel, specify -m parallel or --mode parallel. Events sent on a named channel are delivered to the same named channel in the target correlator. Events sent on the same channel are delivered in order. Events sent on different channels may be re-ordered.
You also specify the -m option when you specify the -x option to disconnect. If you are using a separate connection for each channel you should specify -m parallel when you specify the -x option. If you are using one connection for all channels you should specify -m legacy when you specify the -x option.
See also "Avoid mixing connection modes" later in this section.
-x
When you specify the -x option the behavior depends on whether you also specify the -c option.
If you specify the -x option and you do not also specify the -c channel option then the source correlator stops sending events to the target correlator. Each connection between the source correlator and the target correlator is terminated.
If you specify the -x option and the -c channel option and the utility is using one connection for each channel then the source correlator terminates only the connection(s) it was using for the channel(s) you specify. Any other connections being used for other channels continue to be used. You can specify the -x option with one or more instances of the -c channel option. Remember to also specify -m parallel.
If you specify the -x option and the -c channel option and the utility is using one connection for all channels then the source correlator stops sending events on only the channel(s) you specify. The source correlator continues to send events on any other channels it was already sending events on. If there are no other channels, then the source correlator no longer sends events to the target correlator. However, the connection between the two correlators remains in place. Remember to also specify -m legacy.
-s
Disconnect if slow (only takes effect on first connection)
-f
Read connection information from the named file. If this option is specified, the options –sn –sp –tn –tp -c are all ignored. This file must be in the local character set or in UTF-8 format. If it is UTF-8, specify the -u option in addition to this option.
-u
Indicates that the connection information file is in UTF-8.
-v
Requests verbose output
-V
Displays program name and version number
Operands
None.
Exit status
The following exit values are returned:
Status
Description
0
All connections were established successfully.
1
One or more source correlators could not be contacted
2
One or more target correlators could not be contacted
3
A problem occurred establishing the connection; request invalid
4
Target correlator failed to contact the Source
5
Some other error occurred
Comparison of legacy and parallel connection modes
Legacy connection mode
Parallel connection mode
0 or 1 connection between two correlators.
Any number of connections between correlators.
Events sent on different channels are delivered in the order in which they are sent.
Events sent on different channels may be delivered in a different order from the order in which they were sent.
Sending an event to a named channel delivers the event to the default channel.
Sending an event to a named channel delivers it to only that channel.
Unlike UM for passing events between correlators.
Similar to UM for passing events between correlators.
Same behavior as releases earlier than Apama 5.2.
New behavior starting with Apama 5.2.
UM has no mechanism for enforcing ordering among events sent on different channels. However, UM is the better alternative when you want to use a large number of channels to send events between components. Without UM, the use of two TCP connections with threads on both ends of the connection might reach the limit of how many channels can have dedicated connections.
Avoid mixing connection modes
Successive command lines that specify the same source/target hosts/ports build on each other. While this makes it possible to mix the legacy and parallel connection modes, you should avoid doing that. Mixing connection modes can cause an event to be delivered twice to the same channel. For example:
engine_connect -tp 15902 -sp 15903 -c channelA -c channelB
engine_connect -tp 15902 -sp 15903 -c channelA -c channelC -m legacy
The result of the first command is that there is one (legacy) connection for sending/receiving events on channelA and channelB. The result of the second command is that there is a dedicated connection for sending/receiving events on channelA and a dedicated connection for sending/receiving events on channelC. Events sent on channelA would be delivered twice — once on the legacy connection and once on the dedicated connection.
Examples
Because you can specify command lines that build on each other, you could set up a connection and add named channels later. You can also unsubscribe the channels you've added so that no events are sent or received. The connection remains and you can re-add channels at a later time. However, until you specify the -c option for a given connection, no events emitted by the source correlator are received by the target correlator. Consider the following command line:
engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904
The correlators on host1 and host2 are connected but no channels have been subscribed and therefore no events are sent/received. To send and receive events, specify the -c option as in the following command line:
engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -c CHAN1 -c CHAN2
Now the connected correlators can use CHAN1 and CHAN2 to send/receive events. To add another channel, execute this command:
engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -c CHAN3
The correlators are now using CHAN1, CHAN2, and CHAN3 to send/receive events. To stop using CHAN2, execute the following command. The correlators continue to use CHAN1 and CHAN3.
engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x -c CHAN2
To stop sending and receiving events, execute the following command. Note that the correlators remain connected until one of them stops. There is no penalty for this connection.
engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x
In this example, the following command is equivalent to the previous command.
engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x -c CHAN1 -c CHAN3
Copyright © 2013-2015 Software AG, Darmstadt, Germany and/or Software AG USA Inc., Reston, VA, USA, and/or its subsidiaries and/or its affiliates and/or their licensors.
Use, reproduction, transfer, publication or disclosure is prohibited except as specifically provided for in your License Agreement with Software AG.