Data housekeeping

This section provides you the guidelines and procedures on how to perform housekeeping of your data. Data housekeeping has six aspects, as shown in the following image.

The housekeeping requirements depend on your data size and the number of transactions recorded per day. You need to analyze the existing data and the approximate transactional data that may be recorded on a daily basis. Based on your analysis, you must set the housekeeping parameters. Some of the questions that you need to answer during your analysis are as follows:

What is the data retention period to be set? (for example, 90 days, 120 days, and so on).

Do you need a copy of the older data for long-term retention? (for legal compliance requirements).

Would you ever require restoring data for a particular period? (for forensic analysis).

If you need to export archived data, what would be the storage locations?

Server log considerations. What is the data retention period to be set? (for example, 90 days, 120 days, and so on).

Audit log considerations. What is the data retention period to be set? (for example, 90 days, 120 days, and so on).

Note:
Do not limit your analysis just to these questions. You may have many other considerations based on your organization data structure.

Housekeeping approaches for API transaction data

You can use one of the following approaches to housekeep your API transaction data.

Archive. Archive is the process of moving data that is no longer actively used, to a separate storage location for long-term retention. You may require archived data for future reference, forensic analysis, or regulatory compliance.

Purge. Purge is the process of freeing up space in API data store by deleting obsolete data, not required by the system (data older than the defined retention-period).

The archiving process has a few capacity sizing requirements. The size of the memory and the required storage depends on how much data is stored in every archive interval and the data retention period.

Some of the factors to be considered are as follows:

What is the archive interval? (should you archive once a day, once a week). This factor impacts the memory sizing.

What is the API call volumes? (how many invocations are done per archive interval). This factor impacts memory and disk sizing.

Do you need API payload auditing? (headers, parameters, request, response, and so on). This factor impacts memory and disk sizing.

How long do we keep the archive (daily archives stored for one year). This factor impacts disk sizing.

You must consider other factors based on your organization's requirements.

Purge does not require any additional capacity sizing.

You use a dedicated storage area for archiving that is stored outside the API data store.

Schedule the archive process to be run during non-peak periods as it is generally resource intensive and may affect the performance.

Perform the process of deleting the older archives after the defined retention period (for example after 2 years) outside the product (for example scripts)

The pre-requisites for Purge process are as follows:

For long-term data retention needs, to prevent data loss, you must archive the older data before initiating the purge activity.

CAUTION:
Purge results in irrecoverable loss of data, unless data is archived.

Archive and purge methods

You can automate the archive and purge operations using REST APIs. Alternatively, you can also archive and purge manually from the API Gateway UI. Software AG strongly recommends you to use APIs and automate the process of archive and purge.