Fast Restart (FRS) Disk Compaction Strategies
Since Terracotta is a product intended to maximize performance, a performance-focused "append-only-log" architecture is used for the FRS disk persistence technology.
Terracotta supports the following two compaction strategies:
1. Performance Based Compaction Policy - Favors predictable performance at the expense of unpredictable disk storage space requirements. This is the default strategy.
2. Size Based Compaction Policy - Favors predictable disk storage space requirements at the expense of unpredictable performance.
Overview of Behavior
The following process describes in general what you can expect to observe occurring on the file system in the configured <data> folder. This process is the same, regardless of which of the two compaction strategies you use:
1. Data is added to Terracotta cache(s).
2. An FRS file is created with a file name pattern similar to "seg<nnnn>.frs", where <nnnn> is a number. This file receives all current data writes.
3. As data is further added to Terracotta cache(s), the current FRS file will grow in size.
4. Once the current FRS file grows to some predetermined size, the file is closed.
5. Repeat from Step 2 above, where the file name's <nnnn> is incremented by 1.
It is important to note from the above process that any given FRS file only grows in size to some limit before it is closed and another file is created. This leads to increasing disk consumption.
Based on internal product heuristics, Terracotta will periodically try to "copy" living/valid data into newer FRS files. Once all the cached data in an older FRS file is either "dead" (i.e. the "time to live" has expired) or has been successfully copied to a newer FRS file, then that older FRS file will be deleted in its entirety and its associated disk space will be reclaimed.
In short, disk usage will accumulate to some amount, then some storage space will be "freed", and the process will repeat itself.
Note: It is very important to determine upper-bound used-disk space and ensure the environment always has enough free storage capacity to safely function. We recommend determining this upper-bound, then pad this figure by approximately 20% or more (to accommodate variations in usage), and set that as the "minimum required free disk space" for successful product operation.
Performance Based Compaction Policy (Default)
As noted above, the Performance Based Compaction Policy is the default compaction strategy because it provides the fastest predictable performance which is of paramount concern to "Big Data" users. This strategy frees up disk space associated with data which is either redundant or has been removed from cache. This strategy provides fast, predictable performance at the cost of large (and comparatively cheap) disk storage space. Put another way, this compaction strategy intentionally sacrifices disk space to provide maximal disk performance.
The amount of disk storage space that is required to successfully operate Terracotta when using this compaction strategy is wholly dependent upon a variety of use-case specific factors (such as data size, data volume, configured data lifetime, access patterns, etc.) which cannot reasonably be "guessed at" to determine the minimal FRS disk space requirements for a deployment. The only way to know how much disk space will be required for successful operation is to test the real use-case, under real load, and monitor the resulting disk usage.
The following are some observed real world examples. Please note that every use-case is different and that any particular usage could yield wildly different results; the following is intended to be used for illustrative purposes only to show what is possible as a normally expected disk usage.
Scenario: 50 MB Real Data
FRS was designed to efficiently manage Big Data use-cases; the side effect is that small data volumes typically exhibit a high "relative" disk usage cost.
With this use-case, there are approximately 50 MB of real data which is completely reloaded/refreshed every 24 hours from the backing data store. Based on the data lifetime requirements and access patterns the following behavior might result:
1. Each FRS file slowly grows in size to approximately 512 MB before another file is created.
2. The number of FRS files accumulates to an average of 14 FRS files (each at ~512 MB) before the configured data life-cycle coupled with the data access patterns allows FRS files to be freed.
3. All "old" FRS files are deleted except the most recent 3 FRS files.
Under normal operation, FRS disk usage could grow up to approximately 7.1 GB (14 files of 512 MB each) before 11 FRS files would be deleted (freeing about 5.6 GB) leaving 1.5 GB used on disk. Then the process would repeat, where the 1.5 GB grows to 7.1 GB, at which point 5.6 GB would be freed again. This pattern would repeat over the lifetime of this customer's deployment.
Scenario: 11 GB Real Data
With this use-case, there are approximately 11 GB of real data which is completely reloaded/refreshed every 24 hours from the backing data store. If the used disk-space is observed to peak at approximately 50 GB, we would recommend ensuring that there are 60 GB of free space (50 GB padded by 20%) dedicated to Terracotta data storage.
Scenario: 760 GB Real Data (4 Stripe Terracotta Server Array)
With this use-case, there are approximately 760 GB of real data distributed across 4 Terracotta Servers, with each Terracotta Server containing approximately 190 GB of that data. The majority of the time, this deployment is used for a read-heavy "pure caching" use-case which results in FRS disk usage peaking at about 500 GB on each Terracotta Server. For this usage, it should be safe to ensure that there is 600 GB (500 GB + 20%) of free disk storage capacity reserved for Terracotta usage.
However, once every 2 weeks, a write-heavy "data-reconciliation" process is executed which updates the entire 760 GB data-set (~190 GB per server). Due to the access patterns required to perform the data reconciliation during this bi-weekly process, FRS disk usage temporarily peaks during this window of time on each Terracotta Server at about 2000 GB (2 TB).
In summary, each Terracotta Server:
1. Stores approximately 190 GB of data.
2. Under normal operation requires at least 500 GB of free disk capacity.
3. Only during the bi-weekly data reconciliation operation, the minimum free disk space requirement grows from the normal 500 GB to 2 TB.
Based on the observed application requirements, it is recommended to allocate at least 2.4 TB (2 TB + 20%) of free disk storage capacity to safely operate during this bi-weekly process.
Optional Compaction Strategy (Size Based Compaction Policy)
If constraining FRS on-disk usage is more important than performance (i.e. predictable performance is not important), then the "Size Based Compaction Policy" can be used instead.
This optional compaction strategy is configured in the Terracotta Server's "tc-config.xml" by adding a new "tc-property" to the "tc-properties" section as follows:
By default, this compaction strategy will attempt to constrain the on-disk size to approximately 2 times the overall data size but is influenced by the FRS "segment size".
The default FRS segment file size is 512 MB (53687092 bytes), but this can be further tuned by changing the value used the following "tc-property":
<property name="l2.frs.io.nio.segmentSize" value="53687092"/>
<!-- 512MB = 32 * 1024 * 1024 -->
For small data sets, it might make sense to reduce the FRS segment file size relative to the amount of data being stored; such tuning will be an iterative process is dependent upon the data storage and access requirements of the application.
Configuration Example: 32 MB segment size
Configuration Example: 256 MB segment size
Configuration (Advanced)
The vast majority of use-cases should be successfully handled using the above options; what follows is included for completeness. These additional tuning parameters are only applicable when the Size Based Compaction Policy is being used and should be used with great caution.
Note: The "value" of each parameter is expressed in percent as a fraction of 1.0 such that 100%=1.0, 85%=0.85, 50%=0.5, 5%=0.05, etc.
The following property controls when to trigger/start the on-disk data compaction process. The formula used to determine the trigger threshold is INMEMORY_DATA / DISK_USED.
The following property controls how much data should be compacted at a time. By default, the Size Based Compaction Policy will attempt to free 5% of the on-disk stored data.
<property name="l2.frs.compactor.sizeBased.amount" value="0.05"/>