How Apama integrates with external data sources

You can connect Apama to any event data source, database, messaging infrastructure, or application. There are several ways to do this:

Write transport and codec connectivity plug-ins.

Implement Apama Integration Adapter Framework (IAF) adapters.

Develop custom client applications with Apama APIs for Java, .NET, and C++.

Create applications that use correlator-integrated messaging for JMS.

Connectivity plug-ins can be written in Java or C++, and run inside the correlator process to allow messages to be sent and received to/from external systems. Individual plug-ins are combined together to form chains that define the path of a message, with the correlator host process at one end and an external system or library at the other, and with an optional sequence of message mapping transformations between them.

Connectivity plug-ins perform a similar role to IAF adapters: both allow plug-ins to transform and handle delivery of events. In most cases, we recommend using connectivity plug-ins instead of the IAF for new adapters. The reasons are:

Connectivity plug-ins run inside the correlator process itself. This allows for simpler deployments with less moving pieces. It also avoids problems for handling cases where one of the IAF and correlator are restarted and the other is not, or communication problems (including latency) between them. Ensuring the correct startup order is also simpler (see Sending and receiving events with connectivity plug-ins).

Connectivity plug-ins have a richer data model for both messages and configuration. Connectivity plug-ins can be given events (including sub-events), sequences and dictionaries as map objects. The values within a map can be strings, integers, floats, lists or maps. The lists and maps can in turn contain any of these types (for example, a map can contain a list, where the list contains a map which again contains a list, and so on). This provides an easier to use API for handling nested events and other nested data structures, both in EPL events and in external data formats (such as JSON).

Connectivity plug-ins allow multiple codecs to be used in combination, allowing for a modular approach to message transformation and greater re-use of codecs.

Connectivity plug-ins also perform a similar role to the Apama client library, which allows Java or C++ code in an external process to send/receive messages to/from the correlator. If Apama events need to be made available within an external system, then consider connectivity plug-ins if the external system has a protocol (such as JSON over HTTP). If the external system hosts plug-ins via an API, then the client library may be a better fit.

Apama's Integration Adapter Framework (IAF) provides bidirectional connectivity with event sources and with your environment.

Apama adapters provide both connectivity and XML-based mapping between your application's data format and Apama's internal format. The purpose of an adapter is to translate events from a proprietary format into Apama events. This lets the correlator analyze those events. Also, an adapter converts Apama events into your proprietary source format. This lets Apama send the events to an external service.

Adapters allow a single Apama application to efficiently monitor and analyze disparate event types within a common event processing scenario. For example, the same scenario can process events relating to foreign exchange (FX) aggregation, smart order routing and cross-asset trading in capital markets or cold chain automation in supply chain applications.

Within the IAF, Apama offers a range of standard adapters for capital markets, infrastructure, and connectivity to data and messaging sources, as well as APIs for building custom adapters.

The IAF is a server component that adapters plug into for run-time invocation. You can develop an adapter with the IAF adapter library, along with whatever specific connection APIs you need to connect to your data service. Each adapter is structured so that the mapping of parameters between the source format and Apama format can be configured dynamically through XML.

Middleware messaging adapters. Several middleware bus technologies are available on the market, including technologies by Tibco, IBM MQ Series, Vitria, webMethods, SeeBeyond and others. When the middleware you are using supports JMS, you can create applications that use correlator-integrated messaging for JMS in place of an adapter.

Apama is able to interface with these technologies through an appropriate adapter. If the middleware bus offers publish and subscribe capabilities, then Apama can become a named endpoint like any other service. Apama is able to receive events from the bus and convert them, via an adapter, into Apama events for the correlator to process. An adapter can convert any events emitted by a correlator back into the native bus format.

Database adapters. Apama is able to connect to databases for a number of purposes, including searching historical state or storing key events as an audit trail in the corporate database. Most popular databases support a standard access protocol, such as ODBC or JDBC. The Apama Database Connector (ADBC) provides ODBC and JDBC adapters that use these standard access protocols to connect to your database.

Web Services adapter. The Apama Web Services Client adapter is usable across all Apama components. Interaction with the IAF is transparent to the application.

Custom real-time feed adapters. A number of companies provide real-time content as information feeds. Examples in the finance industry include Reuters, who provides a variety of stock and news feeds, and GLTrade, who provides bidirectional access to a variety of the world's equities and derivatives exchanges. Many such companies use custom communications protocols to provide their data. However, Apama adapters have been easily developed for these and other bidirectional data services.

A range of APIs let you extend Apama at the dashboard, client, and correlator levels for integration with other environments, such as Java, .NET, C, or C++. In addition, you can extend correlator behavior with Java, C and C++ plug-ins that can call external function libraries from within an application.

Apama's correlator-integrated messaging for JMS provides an efficient way to receive and send JMS messages to and from Apama applications. It also provides for reliable messaging (guaranteed delivery) and duplicate detection.