Declared Versus Dynamic In-Memory Stores
Each In-Memory Store holds one dataset with any number of rows, also known as entries, within the memory constraints defined for the store. In previous releases you could store many different datasets in one store with a dataset being one entry in the store.
As shown in the following figure, there are two ways to create In-Memory Stores: declared and dynamic.
Dynamic In-Memory Stores
Dynamic stores are created programmatically on the fly when:
Presto Analytics first stores a dataset to a store that does not yet exist.
By default, Presto Analytics sets the overflowToOffHeap property to true for dynamic stores. All other properties use their default values.
You can set additional configuration properties to use with new dynamic stores created by
Presto Analytics in the
dynamiccache.xml configuration file in the
presto-config/classes folder. For more information on these properties, see
BigMemory documentation.
In addition, each column in the datasets for dynamic In-Memory Stores created by Presto Analytics is defined as searchable with the known or derived datatype for that column. For XML or CSV datasets, columns may all be considered string unless a schema is provided for the dataset. This also uses the Presto Analytics class de.rtm.adapters.bigmemory.RecordMetaDataAttributeExtractor.
An external system creates a store dynamically in
BigMemory. These stores are also known as
dynamic external stores because the external system is also the system the stores data in the store.
Note: | Access to dynamic external stores requires BigMemory be installed as a server or server array. Presto must also have access to the BigMemory license. |
To connect to dynamic external stores, a Presto administrator must define connection configuration. The Terracotta Management Console (TMC) must also be running to successfully connect to a dynamic external in-memory store.
Presto Analytics uses the connection configuration to retrieve configuration for the store from the Terracotta Management Console (TMC) that manages the BigMemory host for this store. This includes search attribute information that is required to allow Presto Analytics to work with the columns in this dataset.
One common example is the use of a dynamic external store to allow Presto to work with datasets from distributed stores in the Apama MemoryStore.
Declared In-Memory Stores
Declared stores are defined in BigMemory configuration files (ehcache.xml files) that Presto administrators add to the Presto Server to allow you to work with these stores. S
Declared In-Memory Stores can contain data loaded by Presto or data loaded by external systems. With declared in-memory stores, the producer system that stores data in the store is the system that creates the store. Presto Analytics is always the consumer system that retrieves data from the store for analysis.
Declaring stores before use allows you to:
Customize store properties for each store, providing better memory management and data retention.
Define search attributes for the columns in the dataset in each store.
Column search attributes are required to allow Presto Analytics to work with datasets stored by external systems. It is optional, but a best practice for datasets stored by Presto Analytics.
Defining search information also allows Presto Analytics to delegate filtering and sorting to the In-Memory Store which provides better performance.
