How Apama integrates with external data sources

You can connect Apama to any event data source, database, messaging infrastructure, or application. There are several ways to do this:

Connectivity plug-ins can be written in Java or C++, and run inside the correlator process to allow messages to be sent and received to/from external systems. Individual plug-ins are combined together to form chains that define the path of a message, with the correlator host process at one end and an external system or library at the other, and with an optional sequence of message mapping transformations between them.

Connectivity plug-ins perform a similar role to IAF adapters: both allow plug-ins to transform and handle delivery of events. In most cases, we recommend using connectivity plug-ins instead of the IAF for new adapters. The reasons are:

Connectivity plug-ins run inside the correlator process itself. This allows for simpler deployments with less moving pieces. It also avoids problems for handling cases where one of the IAF and correlator are restarted and the other is not, or communication problems (including latency) between them. Ensuring the correct startup order is also simpler (see Sending and receiving events with connectivity plug-ins).

Connectivity plug-ins have a richer data model for both messages and configuration. Connectivity plug-ins can be given events (including sub-events), sequences and dictionaries as map objects. The values within a map can be strings, integers, floats, lists or maps. The lists and maps can in turn contain any of these types (for example, a map can contain a list, where the list contains a map which again contains a list, and so on). This provides an easier to use API for handling nested events and other nested data structures, both in EPL events and in external data formats (such as JSON).

Connectivity plug-ins allow multiple codecs to be used in combination, allowing for a modular approach to message transformation and greater re-use of codecs.

Connectivity plug-ins also perform a similar role to the Apama client library, which allows Java or C++ code in an external process to send/receive messages to/from the correlator. If Apama events need to be made available within an external system, then consider connectivity plug-ins if the external system has a protocol (such as JSON over HTTP). If the external system hosts plug-ins via an API, then the client library may be a better fit.

For detailed information, see Using Connectivity Plug-ins.

Apama's Integration Adapter Framework (IAF) provides bidirectional connectivity with event sources and with your environment.

Note:
The IAF architecture is superseded by connectivity plug-ins. Therefore, Software AG strongly recommends choosing connectivity plug-ins over the IAF when creating new adapters and connectivity.

Apama adapters provide both connectivity and XML-based mapping between your application's data format and Apama's internal format. The purpose of an adapter is to translate events from a proprietary format into Apama events. This lets the correlator analyze those events. Also, an adapter converts Apama events into your proprietary source format. This lets Apama send the events to an external service.

Adapters allow a single Apama application to efficiently monitor and analyze disparate event types within a common event processing scenario. For example, the same scenario can process events relating to foreign exchange (FX) aggregation, smart order routing and cross-asset trading in capital markets or cold chain automation in supply chain applications.

Within the IAF, Apama offers a range of standard adapters for capital markets, infrastructure, and connectivity to data and messaging sources (see https://empower.softwareag.com/Products/default.asp), as well as APIs for building custom adapters.

The IAF is a server component that adapters plug into for runtime invocation. You can develop an adapter with the IAF adapter library, along with whatever specific connection APIs you need to connect to your data service. Each adapter is structured so that the mapping of parameters between the source format and Apama format can be configured dynamically through XML.

The following figure shows the bidirectional operation of an adapter.

Examples of types of adapters include:

Middleware messaging adapters. Several middleware bus technologies are available on the market, including technologies by Tibco, IBM MQ Series, Vitria, webMethods, SeeBeyond and others. When the middleware you are using supports JMS, you can create applications that use correlator-integrated messaging for JMS in place of an adapter.

Apama is able to interface with these technologies through an appropriate adapter. If the middleware bus offers publish and subscribe capabilities, then Apama can become a named endpoint like any other service. Apama is able to receive events from the bus and convert them, via an adapter, into Apama events for the correlator to process. An adapter can convert any events emitted by a correlator back into the native bus format.

Database adapters. Apama is able to connect to databases for a number of purposes, including searching historical state or storing key events as an audit trail in the corporate database. Most popular databases support a standard access protocol, such as ODBC or JDBC. The Apama Database Connector (ADBC) provides ODBC and JDBC adapters that use these standard access protocols to connect to your database.

Custom real-time feed adapters. A number of companies provide real-time content as information feeds. Examples in the finance industry include Reuters, who provides a variety of stock and news feeds, and GLTrade, who provides bidirectional access to a variety of the world's equities and derivatives exchanges. Many such companies use custom communications protocols to provide their data. However, Apama adapters have been easily developed for these and other bidirectional data services.

For detailed information, see The Integration Adapter Framework.

A range of APIs let you extend Apama at the dashboard, client, and correlator levels for integration with other environments, such as Java, .NET or C++. In addition, you can extend correlator behavior with Java and C++ plug-ins that can call external function libraries from within an application.

For detailed information, see Developing Custom Clients.

Apama's correlator-integrated messaging for JMS provides an efficient way to receive and send JMS messages to and from Apama applications. It also provides for reliable messaging (guaranteed delivery) and duplicate detection.

For detailed information, see Using the Java Message Service (JMS).

Software AG Digital Event Services is a messaging system for communicating between different Software AG products using events. Digital Event Services allows event definitions to be converted between a product's internal event or document definition to digital event types and vice versa, so participating products can share a set of event definitions. When you develop Apama applications that make use of Digital Event Services, the translation between digital event type definitions and Apama event types is done automatically. When digital events are sent to or received from Digital Event Services, they are converted to or from Apama events.

For detailed information, see The Digital Event Services Transport Connectivity Plug-in.

Apama provides a connectivity plug-in, the MQTT transport, which can be used to communicate between the correlator and an MQTT broker, where the MQTT broker uses topics to filter the messages. MQTT messages can be transformed to and from Apama events by listening for and sending events to channels such as prefix:topic (where the prefix is configurable).

For detailed information, see The MQTT Transport Connectivity Plug-in.

Apama provides a connectivity plug-in, the Kafka transport, which can be used to communicate with the Kafka distributed streaming platform. Kafka messages can be transformed to and from Apama events by listening for and sending events to channels such as prefix:topic (where the prefix is configurable).

For detailed information, see The Kafka Transport Connectivity Plug-in.

Apama provides a connectivity plug-in, the Cumulocity IoT transport, which allows you to communicate with the IoT devices connected to Cumulocity IoT. For example, you can receive events from the devices and send operations to the devices.