Event mappings configuration

The adapter configuration file requires a <mapping> element, which configures the adapter's Semantic Mapper layer.

The <mapping> element may contain the following optional attributes to support the latency framework:

An optional recordTimestamps attribute. The value of the attribute determines the values of the recordUpstream and recordDownstream members of the IAF_TimestampConfig object (C/C++ plug-ins) or TimestampConfig object (Java plug-ins) object passed to the transport or codec. The attribute takes one of the following values:

none — Do not record any timestamps

upstream — Record timestamps for upstream events only

downstream — Record timestamps for downstream events only

both — Record timestamps for both upstream and downstream events

The default, if the recordTimestamps attribute is not present, is none.

An optional logTimestamps attribute. This attribute takes a space- or comma-separated list of keywords for its value. Supported keywords are:

upstream — Log latency for upstream events

downstream — Log latency for downstream events

roundtrip — Log roundtrip latency for all events

log level — Set the logging level for timestamp logging. Any of the standard Apama log levels are accepted for this keyword.

The value of this attribute determines the values of the logUpstream, logDownstream, logRoundtrip and logLevel members of the IAF_TimestampConfig object (C/C++ plug-ins) or TimestampConfig object (Java plug-ins) passed to the transport or codec. If the logTimestamps attribute is not present, the log level defaults to INFO and the other timestamp logging parameters default to false.

The <mapping> element may contain the following (in order):

An optional <logUnmappedDownstream> element, which specifies a file to which unmapped downstream messages should be logged.

An optional <logUnmappedUpstream> element, which specifies a file to which unmapped upstream Apama events should be logged.

One or more <event> elements, specifying the mapping between Apama correlator events and external messages. Setting up the correct <event> elements is the main part of configuring the Semantic Mapper.

One or more <unmapped> elements, which specify events that will bypass the Semantic Mapper, using a string representation of the entire Apama event.

Note:
The order in which <event> and <unmapped> elements appear can be mixed.

Each of these will be discussed in more detail below, after a brief explanation of the operation of the Semantic Mapper.

Semantic Mapper operation

The IAF Semantic Mapper takes as input a set of rules that specify when and how an Apama event can be generated from an external message, and similarly how suitable messages of the correct external format should be constructed from Apama events. These rules are termed an event mapping.

All Apama events must belong to a named event type that defines their structure. On startup, the IAF will parse each <event> element, derive the structure of the event being described, and optionally inject an EPL event definition for it into the correlator. The event mappings are therefore organized by Apama event type, as <event> elements.

When an external message is received from a codec plug-in, the Semantic Mapper will run it past each event mapping sequentially, in the order provided in the configuration file. First it checks whether it matches a set of conditions specified within that mapping. If it does, it proceeds to transform and translate it according to the mapping rules provided. If it does not match the conditions, the Semantic Mapper will move on to the next event mapping. If the message matches against several <event> mappings, only the first mapping is executed unless the breakDownstream attribute is set to false. When this attribute is set to false, all mappings that match are executed.

In the upstream direction, when the Semantic Mapper receives an Apama event, it will already know the type of the event, because this information is part of each event sent out by the correlator. However, it is possible to specify multiple upstream mappings from the same Apama event type; therefore, just as with downstream mappings, the Semantic Mapper will check the incoming Apama event against the conditions defined in each of the mappings for that event type. Just as for downstream mappings, the first matching mapping will be used, unless the breakUpstream attribute is set to false. When this attribute is set to false, all mappings that match are executed.

In both directions it is possible for an incoming event not to match against any event mapping, and in which case no mapping is executed. The <logUnmappedDownstream> and <logUnmappedUpstream> elements allow such messages and events to be logged.

The <logUnmappedDownstream> and <logUnmappedUpstream> elements

These optional elements enable the logging of Apama events and codec messages that were not matched by any of the configured mapping rules, before they are discarded by the adapter. This can be useful for debugging and diagnostics.

The <logUnmappedDownstream> element turns on logging of messages from an external event source that were not mapped onto an Apama correlator event, after being received from a codec plug-in; the <logUnmappedUpstream> element enables logging of events from the correlator that did not match the conditions necessary for mapping to an external message that could be passed on to a codec plug-in.

Both elements have a single attribute called file, which is used to specify the filename that the log should be written to.

For example:

Note:
Due to buffering of files in the operating system, the contents of the log files on disk may not be complete until the IAF is shutdown or reconfigured.

The <event> element

The <mapping> section contains one or more <event> elements, each of which specifies a mapping between an Apama correlator event type and a kind of external message. Setting up the correct <event> elements is the main part of configuring the Semantic Mapper.

Each <event> element can have the following attributes:

name – This is the name of this Apama correlator event type, and is required.

package – This optional attribute specifies the EPL package of the Apama event; if it is not provided, the default package is used. See Developing Apama Applications for information about packages.

direction – This optional attribute defines whether this event mapping is to be used solely for downstream mapping (from incoming external messages to Apama events), upstream mapping (from Apama events to outgoing messages) or for mapping in both directions. The allowed values for the attribute are upstream, downstream and both. The default value if the attribute is undefined is both.

encoder – Required for a mapping that can be used in the upstream direction (direction ="upstream" or "both"), but ignored when processing downstream messages. In the upstream direction the attribute specifies the codec plug-in that should be used to process the message, once the translation process is complete. The name supplied here must match the name provided in the <codec> element.

copyUnmappedToDictionaryPayload – This optional Boolean attribute defines what the Semantic Mapper should do with any fields in the incoming messages that do not match with any field mapping. If copyUnmappedToDictionaryPayload is false, then any unmapped fields are discarded. If it is set to true however, they will be packaged into a special field called __payload, implicitly added as the last field of the Apama event type. Fields in normalized events with a value of null will be included in the dictionary with the value set to an empty string. If this attribute is undefined its value defaults to false.

Using copyUnmappedToDictionaryPayload puts all the payload fields in a standard EPL dictionary that is efficient and easy to access. See The Event Payload for more information about the payload field.

breakUpstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an upstream Apama event to an event mapping.

When set to true, the semantic mapper stops the evaluating process and begins executing the actions specified by that mapping and then goes on to evaluate the next event. This is the default behavior when the breakUpstream attribute is not specified.

When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same event against the other event mappings.

This attribute only affects upstream event processing.

breakDownstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an incoming downstream message to an event mapping.

When set to true, the semantic mapper stops the evaluating process and begins executing the actions specified by that mapping and then goes on to evaluate the next message. This is the default behavior, when the breakDownstream attribute is not specified.

When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same message against the other event mappings.

This attribute only affects downstream event processing.

inject – Determines whether the IAF will automatically inject the event definitions that are implicitly defined by this event mapping into the correlator. The default is false. Injecting events from the configuration files is deprecated. Instead, you should use the -e or --events option of the iaf executable to generate the EPL code for the event definitions and then inject the events (and monitors) during the application's start-up sequence with the tool that you use to start the correlator, such as Software AG Designer or the Apama command-line tools.

copyTimestamps – This is an optional attribute. If set to true (the default is false), the semantic mapper will add an additional field to the generated event definition to hold timestamp information. The new field will be called __timestamps and will be of type dictionary<integer,float> where the dictionary keys are timestamp indexes and the values are the corresponding timestamps. The timestamp field is inserted before the payload field, so it may be the last or next to last field in the event definition. If timestamp copying is enabled for an event type, all timestamps present in the __timestamps field of a matching upstream event will be copied into a new AP_TimestampSet/TimestampSet object and passed to the upstream codec. Likewise, any timestamps passed to the semantic mapper by the codec will be copied into the __timstamps field of the outgoing downstream event and thus made available to the correlator.

transportChannel – optional. If present, then for upstream events (events leaving the correlator), the channel is put in the NormalisedEvent using the value of the transportChannel attribute.

If present, then for downstream events (events going into the correlator), if the value of the transportChannel attribute is in the NormalizedEvent, then that value from the NormalizedEvent is used as the channel name. It is possible that a subsequent <map> element with an identical transport attribute value could override it.

presetChannel – optional. If present, then for downstream events (events going into the correlator), if no channel has been set by the transportChannel attribute, then the value of presetChannel is used as the channel name.

If transportChannel is set, then that value in the NormalisedEvent can still be used for a normal <map> rule, but it will not appear in the unmappedDictionary (if present).

Thus, it is possible to define either a default channel name per type, or a NormalisedEvent field that the transport will send and receive, and this could be re-using a NormalisedEvent field used by a <map> element.

A typical bidirectional <event> element might look like the following. For downstream, if "CHANNEL" (from transportChannel) is in the NormalisedEvent, then the value of the "CHANNEL" entry is used as the channel name, otherwise "channelB" from presetChannel is used. For upstream, the channel name is placed in the "CHANNEL" entry in the NormalisedEvent.

The <id-rules> and <mapping-rules> elements are described below.

The <event> mapping conditions

The <id-rules> element defines a set of conditions that must be satisfied by an incoming message for it to trigger the mapping to an Apama event, or to decide how to map an incoming Apama event back to a normalized message. The <id-rules> element contains <upstream> and <downstream> sub-elements, which in turn contain the mapping conditions to be used when the Semantic Mapper is searching for a mapping to use in the upstream or downstream direction, respectively. Each condition is encoded in an <id> element.

Note:
Conditions are only required for the directions that the mapping can operate in. For example, a mapping with direction="downstream" does not need any <upstream> id rules, while a mapping with direction="both" must specify both <upstream> and <downstream> id rules. The <id-rules>, <upstream> and <downstream> elements themselves must exist though.

<id> - Each <id> sets a condition on a set of fields contained in the normalized message or Apama event. This element takes up to three attributes; fields, which defines the fields that the condition must apply to; test, which specifies the condition; and value, which provides a value to compare the field value with. The value attribute is only required for relational tests. For example:

specifies that the Stock, Exchange and Price fields must exist if the condition is to be satisfied and the mapping proceed. No value is needed to perform the test in this case.

However, the following example:

specifies that the Exchange field must exist and have the value "LSE" for the condition to be satisfied.

Note:
The value for the fields attribute is a list of fields, delimited by spaces or commas. This means, for example that <id fields="Exchange EX,foo" test="==" value="LSE"/> will successfully match a field called "Exchange", "EX" or "foo", but not a field called "Exchange EX,foo". You should keep this in mind when you assign field names for normalized events. While fields with spaces or commas in their names may be included in a payload dictionary in upstream or downstream directions, they cannot be referenced directly in mapping or id rules.

The following test conditions may be specified. The first four tests are unary operators that do not need a value attribute. These tests can be applied to multiple fields in the same <id> rule:

exists - The fields exist, but do not necessarily have a value

notExists - The fields do not exist

anyExists - One or more of the fields exist, not necessarily with values

hasValue - At least one of the fields exists and has a value. To test that multiple fields all exist and all have values, use multiple hasValue conditions, one for each field, such as

The remaining tests are binary relational operators that do require a value attribute. Furthermore, these tests can only be applied to single fields:

== - Case-sensitive string equality

!= - Case-sensitive string inequality

~== - Case-insensitive string equality

~!= - Case-insensitive string inequality

While the equality tests will fail on fields with missing values, the inequality tests will pass.

All <id> conditions in an <id-rules> element must be satisfied for the mapping to proceed.

Note:
A mapping with no <id> elements will always match. This allows a catch-all mapping to be specified. This should be the last definition.

The id rules that test transport field values function in isolation from each other, that is, as soon as a test id rule fails, the mapper stops looking at subsequent rules. This means there is no way to group together tests against the same field name with an OR condition to see if any of them match. Any type of OR value testing needs to be implemented at the codec or EPL layers, or by creating copies of the entire <event> element for each value to test.

The following is an example of a valid <id-rules> element for a bidirectional mapping. Note that the upstream rules are empty, so this mapping will match any incoming Apama event of the appropriate type:

<id-rules>
<downstream>
<id fields="Stock, Price" test="exists"/>
<id fields="Exchange" test="==" value="LSE"/>
</downstream>
<upstream/>
</id-rules>

The <event> mapping rules

The <mapping-rules> element defines a set of mappings that describe how to create an Apama event from an incoming message. Conversely they define the mapping from an Apama event to an outgoing message. Each mapping must be defined in a <map> element, which has the following attributes:

apama – This is the Apama event field name to copy the value into (downstream) or to take the value from (upstream). This attribute is optional. In an upstream direction, if the apama attribute is not specified or is provided empty, a field will be created within the external message and set to the value specified by default. Not specifying the apama attribute has no significance in a downstream direction — this line of the mapping will be ignored.

Note:
The IAF does not know what types are injected into the correlator, and will drop events with a Failed to parse warning if the types and order of the elements with an apama= attribute do not match the event definition that was injected into the correlator. Mapping rules that do not specify an apama= attribute are not affected by this.

transport – This defines the external message's field name to copy the value from (downstream) or to copy the value into (upstream). For a downstream mapping it is possible to define more than one value here; in which case the first encountered is used. This attribute is optional. In a downstream direction, if the transport attribute is not specified or is provided empty, a field will be created within the Apama event and set to the value specified by default. Not specifying the transport attribute has no significance in an upstream direction — this line of the mapping will be ignored.

type – The type of the field in the Apama event type. Any simple correlator type is valid here (string, integer, decimal, float, boolean and location); for complex correlator reference types such as sequence<...> and dictionary<...,...>, specify reference instead. If a field is of reference type, the referenceType attribute can be supplied if needed to define the type (see below). When a field is of reference type, the Semantic Checker passes its string form to and from the codec untouched, and performs no checking upon the validity of the value. Note that fields in the external message are always un-typed character strings, regardless of any type they may have had on the external transport that produced them. Furthermore, the Semantic Mapper does not perform any type "casting" or "coercion" when converting a character string in an external event field to the appropriate Apama type, meaning that the Apama event produced might be invalid and be rejected by the correlator. Conversions in the upstream direction, from the Apama field to a string in the external event, will always succeed. Codec and transport plug-ins should be aware of these rules when working with events that will be, or have been, processed by the Semantic Mapper.

referenceType – This is an optional attribute, but it must be supplied if the attribute type="reference" and the event of which this field is a member has the attribute inject="true" (which is now deprecated) or if the IAF will be run with the -e option in order to generate a EPL file with event definitions. This EPL file is then injected to the correlator (this is the recommended method of injecting events). The value of this attribute must be a valid correlator type. This attribute is only used for the process of constructing the event definitions that are to be injected into a correlator. Note that since this is an XML attribute value, some characters such as angle brackets or quotation marks must be correctly encoded using their XML entity name. For example, a sequence<string> must be written as referenceType="sequence <string>". Note also that when nesting sequences within sequences, a space must be present between the angle brackets to prevent the correlator from parsing this as a bitwise shift.

default – The default value to set the Apama field to if the external field specified in transport is missing. Note that the value provided must be of the type specified in type. For example, a valid string is test or "", a valid integer is 0, a valid decimal is 0.0 or 0.0d, a valid float is 0.0, a valid boolean is true or false, and a valid location is (0.0, 0.0, 0.0, 0.0).

defaultIfEmpty – Optionally, sets the default value to assign to the field in the Apama event if the external field specified in transport is present but has no value defined. The same type considerations apply as for the default attribute. If this attribute is not defined the value specified by the default attribute will apply for this condition as well. Bearing in mind angled brackets and quotes have to be written as XML entities (see above), for a nested event you need to write default="InnerEvent("")" if you want the default value to be InnerEvent("").

Note that at least one of apama and transport must be specified in a mapping.

The following is an example of a valid <mapping-rules> element:

<mapping-rules>
  <map apama="stockName" transport="Stock" type="string" default=""/>
  <map apama="stockPrice" transport="Price" type="float" default="0.0"/>
  <map apama="stockVolume" transport="Volume, TradingVolume,
    CombinedVolume" type="float" default="0.0"/>
</mapping-rules>

In a downstream direction, this specifies that the Stock field must be copied over into stockName, Price must be copied into stockPrice, and the first encountered of Volume, TradingVolume or CombinedVolume must be copied into stockVolume. First encountered means the first such instance when the event is parsed left to right.

In an upstream direction, the Apama field values would be copied into external message fields of the names specified. A given field in an upstream message can be generated from several different sources. These are evaluated in the following order:

1. A <map> rule mapping from a named Apama event field to the transport field.

2. A value for the transport field in the event payload. See The Event Payload for more details on using the event payload.

3. Any default value available from a <map> rule with no corresponding Apama event field.

You may have noticed that while an Apama event must always have its full complement of fields defined and with type-valid values, the same is not assumed of external events.

Note:
If multiple upstream mappings for the same Apama type exist, they must all specify all of the fields in the type in the same order, with the same type values.

Tips for writing a codec when using reference types:

A codec is responsible for constructing the string form of any value. This means that if your event contains a sequence<string> then the codec must generate an entry in the normalized event whose value is of the form:

["string value 1", "string value 2", "Value with a \" and backslash \\"]

If the codec generates an event that the correlator cannot parse, the correlator will drop the event and the codec will have no way of knowing. Be careful constructing the event strings.

Similarly, events from the correlator will contain a normalized event entry whose value is the string from of the field's value, as in the example above. The codec is responsible for parsing these strings.

When writing adapters in Java, Apama suggests you use the classes in the com.apama.event.parser package to parse and construct the strings to send to the Semantic Mapper. If you are writing a C/C++ adapter, the corresponding functions for parsing and constructing strings to send to the Semantic Mapper are found in the AP_EventParser.h and AP_EventWriter.h header files.

For more information on the Java classes, see Working with normalized events as well as the Apama Javadoc. For more information on the C/C++ functions, see Codec utilities.

If nesting other events in the fields of an event, caution must be exercised regarding package namespaces. Always use the fully qualified event name when referencing it in the string form. Also always ensure that the correlator has the enclosed event type defined before the enclosing event type.

The <unmapped> element

The <mapping> section may contain one or more <unmapped> elements, each of which specifies a mapping between the string representation of an Apama event type and a normalized event.

Each <unmapped> element can have the following attributes:

name – This optional attribute specifies the name of the Apama correlator event type to match. If omitted, matches all Apama event types.

package – This optional attribute specifies the EPL package of the Apama event. This attribute can be specified only if the name attribute is also supplied.

transport – This attribute is required; it specifies the field in the NormalisedEvent to map to.

breakUpstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an upstream Apama event to an event mapping.

When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same event against the other event mappings.

This attribute only affects upstream event processing.

breakDownstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an incoming downstream message to an event mapping.

When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same message against the other event mappings.

This attribute only affects downstream event processing.

transportChannel – optional. If present, then for upstream events (events leaving the correlator), the channel is put in the NormalisedEvent using the value of the transportChannel attribute.

If transportChannel is set, then that value in the NormalisedEvent can still be used for a normal <map> rule, but it will not appear in the unmappedDictionary (if present).

In the following example, for downstream, if "CHANNEL" (from transportChannel) is in the NormalisedEvent, then the value of the "CHANNEL" entry is used as the channel name, otherwise "channelB" from presetChannel is used. For upstream, the channel name is placed in the "CHANNEL" entry in the NormalisedEvent.

The <unmapped> mapping conditions

An <unmapped> element must have an <id-rules> element that defines a set of conditions an incoming must satisfy in order to trigger the mapping to an Apama event. If the value of the direction attribute of an <unmapped> element is "both" or "downstream," the <id-rules> element must contain a <downstream> sub-element. The <downstream> sub-element contains conditions to be used by the Semantic Mapper when the message is moving downstream direction. Each condition is encoded in an <id> element.

Each <id> sets a condition on a set of fields contained in the normalized message or Apama event. This element takes up to three attributes; fields, which defines the fields that the condition must apply to; test, which specifies the condition; and value, which provides a value to compare the field value with. The value attribute is only required for relational tests.

The <unmapped> entries behave in the same way as <event> entries — the IAF processes <event> and <unmapped> entries in order, translating events with any that match, and ending at the first entry that has breakUpstream or breakDownstream set to true or not specified (they both default to true).