The Message List codec connectivity plug-in
The Message List codec can be used with a service which supports processing a batch of events in a single request by combining multiple requests into a single list. This requires support from the connected service since the Message List codec changes the type of events being sent.
At high event rates, the correlator will produce batches of messages rather than a single message. Some transports, such as the HTTP client, are inherently single-event based and the maximum rate they can send at depends on the speed of processing a single message. The Message List codec can combine a batch of messages being sent from the correlator into a single message whose body is a list of the original messages in the batch. If the destination service supports this, then the whole batch can be delivered in a single round-trip from the transport.
If the service produces replies which are also a list of replies, then the Message List codec splits them up and delivers them back to the correlator as separate messages.
You need a transport or downstream codec which expects the lists produced by the Message List codec as well as a service which supports them. Often this will be by encoding the lists in something like JSON and then using the String codec to produce a binary message for the transport.
To load the Message List codec, an entry such as the following is required in the
connectivityPlugins section of the configuration file (see also
Configuration file for connectivity plug-ins):
messageListCodec:
libraryName: connectivity-message-list-codec
class: MessageListCodec
You then need to add the messageListCodec to your connectivity chain in the appropriate place. A typical use might be for accessing a web service which has been configured to process lists of messages in JSON format. The HTTP client chain for such a service might look like this:
startChains:
webServiceChain:
- apama.eventMap
- mapperCodec:
...
- classifierCodec:
...
- messageListCodec:
metadataMode: first
- jsonCodec
- stringCodec
- httpClient:
host: ${WEBSERVICE_HOST}
port: ${WEBSERVICE_PORT}
With the above example, the lists produced by the Message List codec are being encoded into JSON to be consumed by the web service.
The following configuration options are available for the Message List codec:
Configuration option | Description |
maxBatchSize | Optional. The maximum number of events that are to be combined into a single list. The actual number will depend on the correlator's accumulation of messages to send. This must be a positive integer. Default: 1000. |
metadataMode | Required. The strategy for handling the metadata of multiple requests. This must be one of the following:  first - Just use the metadata from the first message in the batch as the metadata for the list message. splitBatch - Only add items whose metadata is identical from the batch to a list. Create a new list message when the metadata changes. requestIdList - Use the metadata from the first message, but set metadata.requestId to be a list containing the requestId of each message member - Instead of creating a list of payloads, create a list of maps, each with two members, where metadata refers to the metadata for that message and payload refers to the payload of that message. When converting a list back to separate messages, the above mapping is performed in reverse to create the individual messages. |
The main choice to make is how to handle the metadata when combining multiple messages into a single message. Which choice you will make depends on your application. Let us assume that we have a batch of messages of the following form:
metadata = { requestId: 5, http: { method: PUT, path: /add } }
payload = { name: "Matt", age: 30 }
The payload values and the requestId vary with each message. The examples below show how the Message List codec combines two messages in a batch using the different metadataMode strategies.
metadataMode: first
metadata = { requestId: 5, http: { method: PUT, path: /add } }
payload = [ { name: "Matt", age: 30 }, { name: "James", age: 21 } ]
metadataMode: requestIdList
metadata = { requestId: [5, 6], http: { method: PUT, path: /add } }
payload = [ { name: "Matt", age: 30 }, { name: "James", age: 21 } ]
metadataMode: splitBatch
metadata = { requestId: 5, http: { method: PUT, path: /add } }
payload = [ { name: "Matt", age: 30 } ]
metadata = { requestId: 6, http: { method: PUT, path: /add } }
payload = [ { name: "James", age: 21 } ]
metadataMode: member
metadata = { }
payload = [
{
metadata: { requestId: 5, http: { method: PUT, path: /add } },
payload: { name: "Matt", age: 30 }
},
{
metadata: { requestId: 6, http: { method: PUT, path: /add } },
payload: { name: "James", age: 21 }
}
]
You need to construct your application and the web service it is calling for the strategy that you have chosen. In some cases, you may need an additional Mapper codec to set some of the metadata for the combined message on the transport side of the Message List codec.