Archive and purge using API

You can use the APIs provided by API Gateway to archive and purge data. This method is easy and helps you automate the archive and purge operations.

Transaction and audit data are important. You can archive outdated data (for example, data that is older than a year) for forensic analysis and decision making. Archived data is not the same as Backup data. Software AG recommends that you use a proper naming convention for every archive that you create. You can perform an archive for a specific period (like from 1 June to 1 July).

You can use the following REST API to archive data:

curl -X POST -u "Administrator:manage" -H "content-type:application/json" -H "Accept:application/json"
"http://localhost:5555/rest/apigateway/apitransactions/archives?"time interval"

The following API archives data from 3 June 2021 to 4 June 2021.

curl -X POST -u "Administrator:manage" -H "content-type:application/json" -H "Accept:application/json"
"http://localhost:5555/rest/apigateway/apitransactions/archives?
from=2021-06-12%2000:00:00&until=2021-06-13%2000:00:00&eventType=ALL"

You can schedule archiving using cron jobs or any other scheduling methods. Archiving is a resource-intensive operation. You must not schedule it during peak hours.

You can monitor the status of an archive job using the following API.

curl -X GET -u "Administrator:manage" -H "content-type:application/json" -H "Accept:application/json"
"http://localhost:5555/rest/apigateway/apitransactions/jobs/9c0eefde-dc26-4cb7-b0eb-dfe8f3a8a545"

If the archive job fails, the status field in the above output, displays Failed. You must configure alerts for failures to get notified about the failed archive jobs. Common reasons for failure include health of Elasticsearch cluster, load on the system, and so on. You can look into server logs and analyze the failure reasons.

You can schedule and automate the purge process. You can also purge data manually through the API Gateway UI. To learn more about how to purge data manually, refer to the Archive and purge using UI section. You can purge the following data using commands.

You can purge analytics data based on timeline or size. As an example of timeline based purging, you can purge data older than an year. As an example of size-based purging, you can purge data greater than 100 GB.

Timeline based purging

You can use the following general API to purge analytics data based on time:

curl -X DELETE -u "Administrator:manage" -H "Accept:application/json"
"http://localhost:5555/rest/apigateway/apitransactions/purgeAPITransactions?
action=purge&eventType=eventtype&olderThan=timeline"

In the above API, you can use only the required events in the eventType field. The available event types are transactionalEvents, lifecycleEvents, performanceMetrics, monitorEvents, threatProtectionEvents, policyViolationEvents, errorEvents, auditlogs, applicationlogs.

The olderthan field is the timeline field and can have one of the following values:

Timeline Name	Syntax	Example	Usage
Year	<number>Y	<1>Year	Purges all data up to last 1 year
Month	<number> M	<1> M	Purges all data up to last 1 month
Days	<number>d	<1>d	Purges all data up to last day
Time	<number>h<number> m<number>s	14h30m2s\	Purge all data up to the given time

You can follow one of the following methods to purge data based on size:

You must first identify the data range that needs to be purged, to get your system back to the desired limit. You must query the data for a given date range, determine the size, compare with the requirement and then take the next action (purge or to increase/decrease the date range).

You must define the size limit for indexes that are expected to grow. You must employ the roll-over procedure once the index size exceeds the defined limit. After a roll-over a new index is created. For example, if you have set maximum size for analytics data as 300 GB, maximum size of an index to be 25 GB, and if your data grows to 325 GB, then you have 13 indexes and the size of each index is 25 GB. You can now delete the oldest index so that you get 25 GB of free space and you stay within the defined limits for the analytics data.

Perform the following steps to rollover an index:

Note:
The above API returns the index in descending order of index name. API Gateway follows the pattern gateway_default_transactionalevents_YYYYMMDDHHMM and hence this API returns the index such that the oldest index appears last.

If the purge is successful, you get the following output.

It is essential to backup snapshots that are older than the data retention period. You can delete a backup snapshot using the following command:

To delete backup snapshots that are older than a specific period, you can use the olderthan argument, as shown below:

You must schedule the execution of these commands through cron jobs or other scheduling options. You can schedule the above purging commands of backup to execute once, everyday. You can use the following command to monitor the status of the backup:

apigatewayUtil.sh status backup -tenant default or configured tenant name -name
name of the backup to delete -repoName repo_name

The above command returns success, if the specified name typed in the name field does not exist. If the above command fails,

API Gateway may have empty indexes due to roll-over and purge operations. It is essential to cleanup the empty indexes. You can delete an index if there are multiple indexes and the index to be deleted is not a write index. Software AG recommends that you schedule or perform a backup or archive and then purge the indexes.

You can use the following API to check the documents stored by the indexes.

curl -X GET "http://localhost:9240/_cat/indexes/
gateway_default_analytics_transactionalevents*?s=docs.count:asc&h=i,docs.count"

The above API returns the following response:

gateway_default_analytics_transactionalevents_202106111722 0
gateway_default_analytics_transactionalevents_1622725174422-000001 2

If an index's response value is more than 0, it implies that index is not empty and must not be deleted.

You can use the following API to check if the indexes are write index:

curl -X GET "http://localhost:9240/_cat/aliases/gateway_default_analytics_transactionalevents?
h=i,is_write_index&s=is_write_index:asc"

The above API returns the following response:

gateway_default_analytics_transactionalevents_1622725174422-000001 false
# gateway_default_analytics_transactionalevents_202106111722 true

If an index has a value true, it implies that the index is a write index and should not be deleted.

You can use the following API to delete an index:

You can schedule the purge operation of indexes using a cron job or some other scheduling method. You can schedule index purging on a daily basis. You can monitor the index delete index by using the following API:

If the deletion is successful, the above API returns status code 404. You must configure alerts for failed purge jobs. When a purge job fails, you must check the Elasticsearch logs to troubleshoot.

You can use the following API to delete expired OAuth tokens.

You can schedule the purge operation of indexes using a cron job or some other scheduling method. You can schedule OAuth token purging on a daily basis. You must configure alerts for failed purge jobs. When a purge job fails, you must check the server logs.

You must delete the archive data after it reaches the maximum retention period. There is no API to clear the archive data. You must delete archives manually. You can delete archives on a daily basis.