Managing Runtime Event Store Through Indexes and Aggregates

Runtime events can accumulate in a large volume of data in a very short time. With the size of database increasing exponentially, being able to query the large amount of collected runtime data and return results quickly with minimal time is crucially important. Also, when retrieving, purging, and archiving old runtime data from the database, some queries might take a long period of time to execute causing inconvenience to the customers.

To accelerate the query time for runtime data, CentraSite offers the Customer index and Aggregate data capabilities.

Consumer Index

The Customer index makes queries run faster and more efficiently. The Customer index results in a improved query performance compared to using no index.

Customer index is defined for the collection RuntimeEvents, which collects and stores event data in a database collection within the Event Receiver. The Consumer index field gets automatically filled with the event data collection. The index, however, can only be established if the runtime storage mode is flat. Queries that want to make use of that faster search mechanism must use the new index explicitly. This is, query expressions as

To make your queries run much faster, you must use the Consumer index explicitly.

To use the Consumer index, you must manually update the query expression:

declare namespace re='http://namespaces.CentraSite.com/Schema/SOALink'
let $consumerKey = “ … “
let $eventsForConsumer := collection('RuntimeEvents')/re:events[re:eventsDetails
[re:attributeKey = 'uddi_a7476ff0-a108-11dd-9c38-d8fd010529cc' and re:attributeValue
= $consumerKey]]
…

After you make the required changes, the query expression would look like the following:

declare namespace re='http://namespaces.CentraSite.com/Schema/SOALink'
let $consumerKey = “ … “
let $eventsForConsumer := collection('RuntimeEvents')/re:events[?]
…

Note:
The Consumer index mechanism will not retrieve event data if the value of the attribute $consumerKey is empty, for example, $consumerKey = . However, the mechanism will relieve event data if the value of the attribute $consumerKey is set to unknown.

Aggregate Data

If the acceleration in query performance using the Consumer index does not satisfy and further purging is not possible, you can base queries on aggregates of event data. CentraSite aggregates data by default.

There are two types of aggregate event data:

Event data, which reports the frequency count of events every hour.

Event data, which reports the frequency count of events every day.

Note:
The reporting aggregate event data for every hour and every day depends on the local timezone.

Each aggregate data contains a collection of events within the specified time period concerning a specific triplet of service, target, and consumer.

To make use of the aggregates, event queries can be redefined to be based upon aggregates instead of the original runtime data. Redefining such a query means to read documents of the doctypes rea:aggregateEvents1 (for aggregates on an hourly basis) and rea:aggregateEvents2 (for aggregates on an hourly basis) instead of documents of the doctype events, all in the collection RuntimeEvents. Instead of collecting events within a certain time period, aggregates are summed and totally contained within the time period. Events that are not included by these aggregates are added separately.

To use the aggregate data on an hourly basis, you must manually update the query expression:

declare namespace re='http://namespaces.CentraSite.com/Schema/SOALink';
…
let $events := collection('RuntimeEvents')/re:events [re:timestamp ge $startTime
and re:timestamp lt $endTime]

The above query expression should be replaced to look like the following:

declare namespace re='http://namespaces.CentraSite.com/Schema/SOALink';
…
let $aggregatedEvents := collection(‘RuntimeEvents’)/re: aggregateEvents1
[@re:creationTime ge $startTime and @re:creationTime lt $endTime]
$firstHour := xs:dateTime(concat(substring-before($fromString,":"),":00:00")) +
xs:dayTimeDuration('PT1H'),
$eventsBefore := collection('RuntimeEvents')/re:events [re:timestamp ge
$startTime and re:timestamp lt xs:dateTime($firstHour)]
$lastHour := xs:dateTime(concat(substring-before(string($to),":"),":00:00")),
$eventsAfter := collection('RuntimeEvents')/re:events [re:timestamp ge
$lastHour and re:timestamp lt ,$to]
return ($eventsBefore, $aggregatedEvents, $eventsAfter)

To use the aggregate data on a daily basis, the query needs to be more complex, since hours up to the first complete day within the specified time period, complete days within the time period, and hours after the last complete day up to the end-time requires to be added up to a fixed sum. This would make the query look like the following:

declare namespace re='http://namespaces.CentraSite.com/Schema/SOALink';
…
declare function local:aggregateEvents($fromString as xs:string,
$toString as xs:string) {
let $from := xs:dateTime($fromString),
$to := xs:dateTime($toString),
$firstDay := xs:date(substring-before($fromString,"T"))
+ xs:dayTimeDuration("P1D"),
$lastDay := xs:date(substring-before(string($to),"T")),
$firstHour := xs:dateTime(concat(
substring-before($fromString,":"),":00:00"))
+ xs:dayTimeDuration('PT1H'),
$lastHour := xs:dateTime(concat(
substring-before(string($to),":"),":00:00")),
$eventsBefore := collection('RuntimeEvents')/re:events [re:timestamp ge
$from and re:timestamp lt
xs:dateTime($firstHour)]
$hoursBefore := collection(‘RuntimeEvents’)/re:
aggregateEvents1[@re:creationTime ge $firstHour and
@re:creationTime lt xs:dateTime($firstDay)]
$eventsInDays := collection(‘RuntimeEvents’)/re:
aggregateEvents2[@re:creationTime ge $firstDay and
@re:creationTime lt $lastDay]
$hoursAfter := collection(‘RuntimeEvents’)/re:
aggregateEvents1[@re:creationTime ge xs:dateTime($lastDay) and
@re:creationTime lt $lastHour]
$eventsAfter := collection('RuntimeEvents')/re:events
[re:timestamp ge $lastHour and re:timestamp lt ,$to]
return if ($to lt $from)
then ()
else ($eventsBefore, $hoursBefore, $eventsInDays,
$hoursAfter, $eventsAfter)
}

You can change the data aggregation settings by adding or modifying the attribute EventProcessingSchedulingSettings that is listed below GUIConfiguration in the centrasite.xml configuration file. You can find the centrasite.xml on <CentraSiteInstall_Directory>\cast\cswebapps\BusinessUI\custom\conf.

Default setting for the attribute EventProcessingSchedulingSettings is:

This default setting denotes that aggregates are built for hours and days.

Important:
Be aware that changing the values of this default setting will lead to incorrect search results for queries that combine aggregates produced with different aggregation settings.