Log Aggregation Flavors
API Gateway uses Filebeat to monitor the log files and forwards them to either API Data Store or external ElasticSearch. Filebeat is shipped along with API Gateway and its configuration can be found in the following location:
SAG_Install_Dir/profiles/IS_Instance_Name/apigateway/filebeat/filebeat_apigateway.yml
When log aggregation is enabled, API Gateway initiates the Filebeat process in the background to monitor the log files and forwards the log files to either API Data Store or external Elasticsearch.
The following are some of scenarios that API Gateway supports for log aggregation:
API Gateway + API Data Store
In this approach the aggregated logs are stored in the API Data Store. And if you use the same API Data Store to store the API Gateway metadata such as APIs, policies, configurations, run-time events, and so on. The API Data Store might be overloaded with too many requests and data might grow exponentially. Hence, it is important to keep track of the log aggregation data and purge them periodically.
API Gateway + External Elasticsearch
In this approach the aggregated logs are stored in an external Elasticsearch. This is the recommended approach for production environment so that you does not have any overhead on the API Data Store when compared to the API Gateway + API Data Store approach. But you must create your own dashboard to view the logs and you must come up with your own logic for archiving and purging the aggregated logs.
API Gateway + API Data Store + Terracotta Server
You can also aggregate Terracotta server logs. The Terracotta server can be installed in following ways:
If Terracotta Server and API Gateway is installed in same machine, perform the following steps:
1. By default, the Terracotta logs are available in the (user.home)/terracotta/server-logs location. You can change this folder by specifying the custom log data location in the tc-config.xml. For more information, see Terracotta documentation.
2. In the Administration > General > Application logs > Log aggregation section, disable the log aggregation.
3. Update the filebeat_template.yml file in the SAG_Install_Dir\profiles\IS_Instance_Name\apigateway\filebeat location with the following content. Make sure you place the following content below the DashboardLogs configuration.
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:/Users/alice/terracotta/server-logs/terracotta-server.log
fields:
node: ${NODE}
fileType: TerracottaServerLogs
timezone: ${TIMEZONE}
fields_under_root: true
multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
multiline.negate: true
multiline.match: after
4. In the Administration > General > Application logs > Log aggregation section, enable the log aggregation.
5. In the Analytics > Application Logs section, check whether the Terracotta Server logs are aggregated.
If Terracotta Server and API Gateway are installed in different machine, perform the following steps:
1. Edit the tc-config.xml in Terracotta Server with the following content to store the logs in a network location and start the Terracotta Server.
<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
<servers>
<server host="localhost" name="%h">
<data>C:/install/TerraCottaServer/Terracotta/server/bin/data</data>
<logs>//worstation01/share/tcserverlog</logs>
<offheap>
<enabled>true</enabled>
<maxDataSize>512m</maxDataSize>
</offheap>
</server>
<restartable enabled="true"/>
</servers>
</tc:tc-config>
2. In the Administration > General > Application logs > Log aggregation section, disable the log aggregation.
3. Update the filebeat_template.yml file in the SAG_Install_Dir\profiles\IS_Instance_Name\apigateway\filebeat location with the following content. Make sure you place the following content below the DashboardLogs configuration.
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- //mckmut02/share/tc-server-logs/terracotta-server.log
fields:
node: ${NODE}
fileType: TerracottaServerLogs
timezone: ${TIMEZONE}
fields_under_root: true
multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
multiline.negate: true
multiline.match: after
4. In the Administration > General > Application logs > Log aggregation section, enable the log aggregation.
5. In the Analytics > Application Logs section, check whether the Terracotta Server logs are aggregated.
Configuring Log for Elasticsearch Client in API Gateway
API Gateway uses Elasticsearch REST client to connect to API Data Store or External Elasticsearch. By Default, the logs that are created by these REST clients are ignored to avoid over logging. To trouble shoot or diagnose elasticsearch calls from API Gateway, we can manually configure the log configuration for Elasticsearch REST client by following the below steps:
1. Open the log4j2.properties located at SAG_Install_Dir\profiles\IS_Instance_Name\configuration\logging.
2. Add the following configuration at the end of the file.
logger.10.name=org.elasticsearch.client
logger.10.additivity=false
logger.10.level=info
logger.10.appenderRef.lar.ref=ESRestClient
logger.11.name=org.apache.http
logger.11.additivity=false
logger.11.level=info
logger.11.appenderRef.lar.ref=ESRestClient
logger.12.name=org.apache.http.wire
logger.12.additivity=false
logger.12.level=info
logger.12.appenderRef.lar.ref=ESRestClient
logger.13.name=org.apache.http.impl.conn
logger.13.additivity=false
logger.13.level=info
logger.13.appenderRef.lar.ref=ESRestClient
appender.esrestclient.name=ESRestClient
appender.esrestclient.type=RollingFile
appender.esrestclient.fileName=<INSTALL_LOCATION>/IntegrationServer/instances/default/logs/ESRestClient.log
appender.esrestclient.filePattern=<INSTALL_LOCATION>/IntegrationServer/instances/default/logsESRestClient.log.%i
appender.esrestclient.layout.type=PatternLayout
appender.esrestclient.layout.pattern=%d [%t] %-5p %c %x - %m%n
appender.esrestclient.policies.type=Policies
appender.esrestclient.policies.size.type=SizeBasedTriggeringPolicy
appender.esrestclient.policies.size.size=10MB
appender.esrestclient.strategy.type=DefaultRolloverStrategy
appender.esrestclient.strategy.max=10
#This is a custom Platform provided filter which matches all log messages that contain OSGi data in MDC (bundle.id, component.name, etc.)
appender.esrestclient.filter.osgi.type=LogServiceFilter
appender.esrestclient.filter.osgi.onMatch=DENY
appender.esrestclient.filter.osgi.onMismatch=NEUTRAL
3. Restart API Gateway.
You can see the ESRestClient.log at the SAG_Install_Dir\IntegrationServer\instances\instance_name\logs location. You can see the request sent to particular elasticsearch node and what is the response status code in the ESRestClient.log