Troubleshooting Tips: API Gateway and DMZ Connectivity
I see several requests waiting for a registration connection
This might occur when the network and connections are unreliable.
On the internal server (Green Zone), configure the property watt.server.rg.internalregistration.timeout that controls how long the API Gateway server waits for a connection to the internal server before closing an unresponsive connection to the API Gateway server.
To configure this property:
1. Navigate to User menu > Administration > General > Extended settings.
2. Click Show and hide keys.
3. Select watt.server.rg.internalregistration.timeout property in the watt properties section.
4. Provide a suitable value as follows:
Provide a value 0 when the network and connections are reliable, so that the connection between internal server and API Gateway never times out. This is the default value of this property.
Provide a non-zero value when the network and connections are unreliable or when the connections breakdown. This value specifies the time after which API Gateway stops trying for a unresponsive internal server connection so that the connections time out.
5. Click Save.
Considerations while configuring the watt.server.rg.internalregistration.timeout property:
When you set the property to a value within which if API Gateway does not receive any requests from DMZ, then registration internal ports are auto refreshed. When the connectivity between API Gateway and DMZ is broken after a refresh (disabled and enabled), set the internal ports manually to resolve the issue.
When you set the property to a value that is lower than
watt.net.socketpool.sweeperInterval, the internal server closes the connection to the API Gateway server and re-establishes a new connection regularly.
The property watt.net.socketpool.sweeperInterval specifies the frequency, in seconds, at which the socket pool sweeper performs. The socket pool sweeper sends a ping request to all API Gateway connections and HTTP client connections. During a sweep it removes any invalid HTTP client connections. By default, the sweeper sends a ping request every 60 seconds.
As a good practice, Software AG recommends enabling the watt.net.socketpool.sweeperInterval setting, if you are using the watt.server.rg.internalregistration.timeout property. Set the value of watt.server.rg.internalregistration.timeout on the internal server to a value greater than the ping values defined by the watt.net.socketpool.sweeperInterval server configuration parameter on the API Gateway server.
In addition to the above parameter setting also increase the connection from internal server by ensuring that you have enough threads configured in both DMZ and internal server to handle the load.
I see client requests on the API Gateway server (DMZ) waiting indefinitely for a connection to the internal server (Green zone).
This might occur when the connections breakdown.
Configure the property watt.server.rg.internalsocket.timeout that controls how long the API Gateway server waits for a connection to the internal server and returns a HTTP 500-Internal Server Error to the requesting client.
To configure this property:
1. Navigate to User menu > Administration > General > Extended settings.
2. Click Show and hide keys.
3. Select watt.server.rg.internalsocket.timeout property in the watt properties section.
4. Provide a suitable value.
If a connection to the internal server becomes available within the specified timeout period, API Gateway server forwards the request to the internal server.
If a connection does not become available before the timeout elapses, API Gateway server returns a HTTP 500-Internal Server error to the requesting client.
5. Click Save.
I see client authentication is not enforced for APIs invoked on API Gateway server (DMZ) if requests are sent to the external port.
For an API invocation on API Gateway server (DMZ) if requests are sent to the external port you may observe that there is no client authentication performed and you may observe that the enforced IAM policies fail.
To enable client authentication for requests that sent to the external port, you must set the watt.server.revInvoke.proxyMapUserCerts property as follows:
1. Navigate to User menu > Administration > General > Extended settings.
2. Click Show and hide keys.
3. Select watt.server.revInvoke.proxyMapUserCerts property in the watt properties section and set it to true.
I see some of the internal API invocations are processed in the API Gateway server (DMZ) instead of API Gateway server (Green zone) when using an API Gateway Advanced Edition.
This is observed when you have the paired deployment setup and you have enforced only threat protection policies in API Gateway server (DMZ) and all other policies in internal server (Green zone) and you have configured port access restrictions to allow access only to the APIs hosted on the API Gateway (say with /gateway/, /ws/ , and so on). In such a case you must provide access to the following APIs in case the APIs are protected by security policies such as OAuth, OpenId or JWT. Allowing access to these endpoints is important for API Portal and API consumers to access API Gateway to retrieve the tokens.
pub.apigateway.oauth2:getAccessToken
pub.apigateway.oauth2/getAccessToken
pub/apigateway/oauth2/getAccessToken
secure.apigateway.oauth2/approve
secure.apigateway.oauth2:approve
secure/apigateway/oauth2/approve
pub.apigateway.oauth2:authorize
pub.apigateway.oauth2/authorize
pub/apigateway/oauth2/authorize
pub/apigateway/openid/getOpenIDToken
pub/apigateway/openid/openIDCallback
pub/apigateway/jwt/getJsonWebToken
/pub/apigateway/jwt/certs
/pub/apigateway/jwt/configuration
/pub/apigateway/jwt/thirdPartyConfiguration
By default, if API Gateway server (DMZ) has API Gateway Advanced Edition then API requests are processed on the API Gateway server in DMZ but the actual enforcement happens in the internal server. Hence, these API requests must be processed on internal server. To resolve this, configure the extended setting forwardInternalAPIsRequest as follows:
1. Navigate to User menu > Administration > General > Extended settings.
2. Click Show and hide keys.
3. Select the extended setting forwardInternalAPIsRequest and set it to true.
I see that the internal APIs can be accessed through the External Port of API Gateway server deployed in DMZ
API Gateway supports the safe exposure of APIs by featuring threat protection and enforcing identity and access management policies. The reverse invoke capabilities of API Gateway supports the exposure of APIs that are running entirely behind a firewall or where just the service implementing the APIs is protected by a firewall.
In cases where you can access the internal APIs through the external port of API Gateway server in DMZ, you can block requests to internal APIs in API Gateway as follows:
1. Configure the following ports in API Gateway located in DMZ:
5500: API Gateway default HTTP regular port
9200: API Gateway external port (ExtPortAlias)
9201: API Gateway registration port (DefaultRegPortAlias)
9202: API Gateway registration port (ApiRegPortAlias)
With the above configuration, the API Gateway instance in DMZ receives requests on the external port ExtPortAlias. These requests are forwarded to the registration port DefaultRegPortAlias. DefaultRegPortAlias is connected to an API Gateway internal port that is defined on the API Gateway in the DMZ. This ensures that requests are not forwarded to the Integration Server or API Gateway in the green zone, but processed within the API Gateway in DMZ. The default registration port is the first one defined for an external port. There is no specific naming required.
2. Configure the routing policies in the API Gateway instance located in DMZ, as follows.
To forward API requests to the backend services, the API routing policies must point to the ApiRegPortAlias, the second registration port alias defined for the external port.
The endpoint URI of the Straight Through Routing policy leverages the apigateway scheme and references the ApiRegPortAlias. The resource path of the endpoint URI points to the sample flow service employee running on the Integration Server in green zone. This flow service can not be invoked directly from the DMZ external port. All non-API requests are routed to the internal port of the API Gateway in DMZ where the backend flow services are not defined.
3. Configure the internal port of the Integration Server in green zone as follows.
The registration port ApiRegPortAlias is associated with the internal port of the Integration Server in the green zone. The figure depicts the port configuration screen with the internal port connected referencing the registration port ApiRegPortAlias on the API Gateway in DMZ.
My internal server has run out of registration connections
My internal server has run out of registration port and I see the following error message.
number requests waiting for a registration connection.
Resolution
Each connection consumes a thread, either from the API Gateway in green zone's common thread pool or from the internal listener's private thread pool, if one is defined. The consumed thread can only be used to process requests from API Gateway in DMZ.
If you have defined a private thread pool for the internal registration listener, the number of connections you can specify in the Max Connections box is limited to the maximum number of threads allowed in the private thread pool for this listener.
If you have multiple internal registration listeners, each with its own private thread pool, the same rule applies for each internal registration listener.
If you have not defined a private thread pool for an internal registration listener, a reasonable limit for the Max Connections box is 75% of the number of server threads specified in Server Thread Pool Max Threads box on the Settings > Resources page. If you have multiple internal registration listeners and none of them have private thread pools, the sum of all connections specified in the Max Connections boxes for these listeners should not exceed 75% of the number of server threads specified in Server Thread Pool Max Threads.
A thread remains open unless it is closed by a firewall, a network glitch, or an exception.