Concepts

A connectivity chain is used to get events of a certain shape from a particular external source and convert them from a given format into Apama events and back. There will be at least one chain for each event source. There could also be one chain for each format of events you want to receive from that source. There can be multiple instances of a given chain definition in the configuration file, corresponding to multiple connections to that source. The choice of which chain to use for a given event may vary between different sources.

All chains begin with a host plug-in, which defines how events are converted into Apama events. All chains end with a transport, which determines the external source we are connecting to. Between these you have an ordered sequence of any number of codecs. These are used to convert messages from the format produced by the transport to that which the host plug-in will consume and vice versa. Messages are passed between the elements in a connectivity chain using an abstract message format made up of maps, lists, byte arrays and various primitive formats. As well as the (arbitrary) payload, there is also a (nested) map of metadata with string keys and any of the supported types as values.

The following diagram shows a typical connectivity chain:

The connectivity chain in the above diagram uses several plug-ins and codecs:

The host plug-in is the connection between the correlator and the chain of codecs that manipulate and transform the messages as they pass through the chain. There are two host plug-ins to choose from:

apama.eventMap. This is the most common choice for a host plug-in. It produces and consumes Apama events as a map with the keys being the names of the fields in the event and the metadata.sag.type element set to the name of the event.

apama.eventString. This host plug-in consumes events in Apama's proprietary string format. It is usually used for interaction with legacy systems such as the IAF which use Apama's string format.

Any message is transformed into a compatible form for either the host or the transport as it flows through the codec pipeline toward its destination.

The transport is the plug-in that can actually retrieve or send the message to the external resource. This is can be one of the standard plug-ins such MQTT or HTTP (see Standard Connectivity Plug-ins for detailed information) or a plug-in written by the user.

Any message passing through from a transport plug-in to the host has common features that will be similar in most implementations. The following diagram describes the common features (metadata and payload) and indicates some of the common values they can take.

Codecs may be of various types. The following diagram describes an incoming message transformation:

Apama provides a selection of codecs by default for converting from common formats and for use with customer-provided codecs to provide standard transformations. The above diagram uses the following standard codecs:

The String codec transforms an inbound message to an Apama string. It converts a buffer (transport side) to a string (host side) by decoding it as UTF-8.

The outbound flow performs the reverse mapping in the other direction (see the diagram below) and places a string representation of the message into an Apama string.

The JSON codec transforms the inbound message to a compatible representation. It converts a string (transport side) to a structured type (usually map) by parsing it as a string JSON object.

The outbound flow performs the reverse mapping in the other direction (see the diagram below) and transforms a JSON-compatible representation to a JSON string in the payload.

Messages to the host are identified as a particular event type by matching patterns on the metadata or message contents. The Classifier codec defines what the metadata.sag.type will be and uses rules that are triggered by values held by a field or fields in the inbound message.

The Mapper codec sets up the map fields ready for either the correlator or transport depending upon the direction of the flow. It moves fields in the event around and adds default values to missing fields. It can have different rules for different event types, and it can have symmetric rules or different rules depending on direction. towardsHost is the inbound direction, and towardTransport is the outbound direction. The metadata and payload keys provide the source or destination for the values.