The Merge Engine and Event Deltas

In order to streamline web-based Publish/Subscribe applications, it is possible to deliver only the differences between consecutive events, as opposed to the entire event each time. These event deltas minimize the amount of data that needs to be sent from the publisher, as well as the amount of data ultimately delivered to subscribers.

Imagine a channel that is used to deliver foreign-exchange currency prices. Let us assume that the channel has a publish-key named pair, of depth 1, representing the currency pair. This means that a maximum of one event for each currency pair will exist on the channel at any time.

An event representing a foreign-exchange currency price might therefore be published as follows:

var event = Nirvana.createEvent();
var priceDictionary = myEvent.getDictionary();

priceDictionary.putString("pair", "EURUSD");
priceDictionary.putFloat("bid", 1.2261);
priceDictionary.putFloat("offer", 1.2263);
priceDictionary.putFloat("close", 1.2317);
priceDictionary.putFloat("open", 1.2342);

demoChannel.publish(event);

Let us now imagine that the spread on this price has tightened: while the bid value remains the same, the offer is lowered from 1.2263 to 1.2262.

Under normal circumstances, an entire new event would be created and published:

var event = Nirvana.createEvent();
var priceDictionary = myEvent.getDictionary();

priceDictionary.putString("pair", "EURUSD");
priceDictionary.putFloat("bid", 1.2261);
priceDictionary.putFloat("offer", 1.2262);
priceDictionary.putFloat("close", 1.2317);
priceDictionary.putFloat("open", 1.2342);

demoChannel.publish(event);

Notice that the majority of the information in this new event is no different to that in the previously sent event.

Event deltas allow us to publish only the information that has changed:

var event = Nirvana.createEvent();
var priceDictionary = myEvent.getDictionary();

// we need to specify the publish-key too, of course
priceDictionary.putString("pair", "EURUSD");
priceDictionary.putFloat("offer", 1.2262);

event.getAttributes().setAllowMerge(true);

demoChannel.publish(event);

It is clear from the above example that using event delta functionality through setAllowMerge(true) in the Event Attributes object is especially useful when publishing events with many dictionary keys that have unchanged values.

In the above example, where the channel had a publish-key named pair with a depth of 1, only one event for each currency will exist on the channel at any one time. Given that the last published event was a mere delta, how can we guarantee that a new subscriber will receive an event with a fully populated dictionary containing all expected keys?

Universal Messaging's Merge Engine will process and merge events with all event deltas, maintaining internal representations of merged event snapshots, keyed on the channel's publish-key. A merged event snapshot for each unique publish-key value is delivered to subscribers when they initially subscribe, or when they reconnect after a period of disconnection.

Web clients built using the Universal Messaging JavaScript API can receive any combination of standard events, event deltas and merged event snapshots.

A client that subscribed to the channel some time after the above example's event delta was published would receive a server-generated merged event snapshot with a dictionary containing the following key/value pairs:

Note how the offer value of 1.2262 has been merged into the older event's dictionary.

A client that was subscribed before the initial example event was published would receive two events. The first event would have a dictionary containing the following key/value pairs:

The second event received by the client (the delta) would be marked as a delta, and have a dictionary containing only the following key/value pairs:

In summary, therefore, any new client subscribing will receive all of the fields in the merged event for EURUSD, while any existing subscribers will only receive the offer change for EURUSD.

Important: Note that only the event delta is passed to the developer-implemented Channel.on() callback; it is the developer's responsibility to make use of the deltas in this case.

In order for a channel to be capable of delivering deltas and merging events it must be created with the Merge Engine enabled, and it must have a single publish-key. The publish-key represents the primary key for the channel.

If a publisher of an event does not make a call to setAllowMerge(true) then the merged event snapshot for that publish-key value would be replaced in its entirety by the newly published event.

If a subscriber disconnects and then reconnects it will again receive the latest snapshot before receiving only the deltas that are subsequently published.