Google Protocol Buffers

Google Protocol Buffers are a way of efficiently serializing structured data. They are language and platform neutral and have been designed to be easily extensible. The structure of your data is defined once, and then specific serialization and deserialization code is produced specifically to handle your data format efficiently.

Universal Messaging supports server-side filtering of Google Protocol Buffers, and this, coupled with Google Protocol Buffer's space-efficient serialization can be used to reduce the amount of data delivered to a client. If server side filtering is not required, the serialised protocol buffers could be loaded into a normal nConsume Event as the event data.

The structure of the data is defined in a .proto file, messages are constructed from a number of different types of fields and these fields can be required, optional or repeated. Protocol Buffers can also include other Protocol Buffers.

The serialization uses highly efficient encoding to make the serialized data as space efficient as possible, and the custom generated code for each data format allows for rapid serialization and deserialization.

Google supplies libraries for Protocol Buffers in Java, C++ and Python, and third party libraries provide support for many other languages including .NET, Perl, PHP etc. Universal Messaging's client APIs provide support for the construction of Google Protocol Buffer events through which the serialized messages can be passed.

These nProtobufEvents are integrated seamlessly in Universal Messaging, allowing for server-side filtering of Google Protocol Buffer events, which can be sent on resources just like a normal Universal Messaging Event. The server side filtering of messages is achieved by providing the server with a description of the data structures (constructed at the .proto compile time, using the standard protobuf compiler and the --descriptor_set_out option). The default location the server looks in for descriptor files is /plugins/ProtobufDescriptors and this can be configured through the Enterprise Manager. The server will monitor this folder for changes, and the frequency of these updates can be configured through the Enterprise Manager. The server can then extract the key value pairs from the binary Protobuf message, and filter message delivery based on user requirements.

To create an nProtobuf event, simply build your protocol buffer as normal and pass it into the nProtobuf constructor along with the message type used (see the programmatic example below).

Example.Builder example = Example.newBuilder();
example.setEmail("example@email.com");
example.setName("Name");
example.setAddress1("Norton Foldgate");
example.setHouseNumber(1);
byte[] buffer = example.build().toByteArray();

nProtobufEvent evt = new nProtobufEvent(buffer,"example");
myChannel.publish(evt);

nProtobuf events are received by subscribers in the normal way.

public void go(nConsumeEvent evt) {
if (evt instanceof nProtobufEvent) {
totalMsgs++;
// Get the data of the message
byte[] buffer = evt.getEventData();

if(((nProtobufEvent) evt).getTypeName().equals("BidOffer")){
BidOffer bid = null;
bid = BidOffer.parseFrom(buffer);
//......//
}
}
}

The Enterprise Manager can be used to view, edit and republish protocol buffer events, even if the Enterprise Manager is not running on the same machine as the server. The Enterprise Manager is able to parse the protocol buffer message and display the contents, rather than the binary data.

All descriptors will be automatically synced across the cluster if the channel is cluster-wide.

Protocol buffers have their own section in the Enterprise Manager Config panel.

These are explained via their tooltips.

UseChannelLevelProtobufCache indicates whether the descriptors are set against the channel, or put into a folder (see "Legacy Google Protobuf Support" below). UpdateDescriptorInterval, the maximum and minimum builder numbers, and the input and output directories only apply for the legacy option.

FilterProtobufEvents is set to true by default, and must be set to true to enable filtering of protobuf events.

If nested messages need to be filtered on, then GlobalValues -> ExtendedMessageSelectors, must be set to true. Again this is now enabled by default but will not be enabled in installs upgraded from older versions.

When creating a channel via the Enterprise Manager, there is a protobuf descriptor section on the create dialogue. Clicking "Set..." here brings up a file dialogue where a descriptor file (generated when the protobuf is compiled, as described above) can be selected. Multiple descriptor files can currently only be set programmatically, not via the Enterprise Manager.

Any channel with an associated descriptor can be snooped in the normal way. Enterprise Manager will use the descriptor to deserialise the message, and will show the contents of that message in the event details.

//Create a realm node (this is standard administration API connection)
realm = new nRealmNode(new nSessionAttributes(testServer.getDefaultAdapter()));
realm.waitForEntireNameSpace();

//Create a channel with the descriptors.

Path path =Paths.get("../../changeManagement/test/protobuf/SAGTester.fds");

byte[] bytes = Files.readAllBytes(path);
byte[][] descriptors = new byte[1][bytes.length];
descriptors[0]=bytes;
myAttribs.setProtobufDescriptorSets(descriptors);
myChannel = nsession.createChannel(myAttribs);

Then we can publish using the protobuf serialised as usual, along with the "name" of the protobuf message type.

You can then use Universal Messaging style message filters, as you would for normal events. e.g. "Name='test'".

The protocol buffer definition files associated with a store (i.e. a channel or a queue) can be updated without requiring the store to be deleted and re-created. Once updated, all filtering will be done with the new protobuf definitions.

The definitions can be updated either programmatically or via the Enterprise Manager.

To update the protocol buffer definitions for a store, proceed as follows:

3. On the file select page, select the file or files which contain the descriptors you wish to set on this store (multi-select is enabled for loading multiple file descriptor sets). Then click "Open".

Stores have a method updateProtobufDescriptors(byte[][] descriptors). This takes an array of the descriptors to be applied to the channel. After calling this method, the new descriptors will be loaded and will be used for filtering on the channel from that point onwards. The code for performing this can be seen below.

This API is available via the client API. The Admin API is not required in order to perform these operations.

These nProtobufEvents are integrated seamlessly in Universal Messaging, allowing for server-side filtering of Google Protocol Buffer events, which can be sent on resources just like normal Universal Messaging events. The server side filtering of messages is achieved by providing the server with a description of the data structures (constructed at the .proto compile time, using the standard protobuf compiler and the --descriptor_set_out option). The default location the server looks in for descriptor files is /plugins/ProtobufDescriptors and this can be configured through the Enterprise Manager. The server will monitor this folder for changes, and the frequency of these updates can be configured through the Enterprise Manager. The server can then extract the key value pairs from the binary Protobuf message and filter message delivery based on user requirements.

The Enterprise Manager can be used to view, edit and republish protocol buffer events, even if the Enterprise Manager is not running on the same machine as the server. To enable this, the server outputs a descriptor set to a configurable directory (by default the htdocs directory for the realm) and this can then be made available through a file plugin etc. The directory can be changed through the Enterprise Manager. The Enterprise Manager can then be configured to load this file using -DProtobufDescSetURL and then the contents of the protocol buffers can be parsed.