Retry and Recovery Architecture

A highly available and reliable system has the ability to recover from temporary errors. If a transient error is encountered during execution, then the system must perform the following tasks:

The first phase in recovery is defining a transient error. For a webMethods adapter, a transient error goes away in time, without requiring human intervention on Integration Server. For example:

Non-transient error. If an adapter service performing an insert is called, and the backend resource rejects the insert because the data format is incorrect, then the error is not transient. The data must be manually reformatted for the insert to complete correctly.

Transient Error. If an adapter service performing an insert is called, and the backend resource is offline for scheduled maintenance, then the insert fails. However, when the system comes back on line, the insert works. Retrying the operation in this scenario is useful.

The ability to recognize these errors is critical in every component of an adapter. It is up to the adapter developer to determine which backend errors are transient. When an adapter recognizes this situation, it throws an AdapterConnectionException to alert Integration Server about the error. The AdapterConnectionException is a subclass of the ResourceException. AdapterConnectionException is a valid exception from any ADK method that declares a throws ResourceException.

An adapter developer must isolate the errors that are transient. When Integration Server catches the AdapterConnectionException, then Integration Server initiates the retry operations. If Integration Server initiates the retry operations for non-transient errors, then the processing time is wasted.

Transient exceptions within adapters are related to connection errors. Clear and regenerate the connection used in the operation before retrying.

Based on the AdapterConnectionException received from the adapter, Integration Server cleans the entire pool or just the current connection. By default, the exception is an indication to clean the entire pool. For most applications this is the correct behavior.

Notifying the connection pool manager, and destroying and discarding all the free connections cleans the pool.

The regeneration occurs when a new connection is requested. If the connection is not pooled, an attempt is made to create the connection. If successful, a connection is created and used. If the connection is pooled and only the single connection is destroyed, another connection is reserved from the pool and used. If the pool is cleared, the pool attempts to fill to the minimum number of connections specified for the pool. If successful, a connection is reserved and used.

If the transient error occurs at any stage of the pool regeneration or connection creation, the adapter must throw an AdapterConnectionException, that ends the current retry operation. Then Integration Server starts another retry operation or ends the retry loop if the maximum attempts are made.

The retry facility is provided by Integration Server. The retry facility depends on the adapter construct that detected the failure. A retry is configured with a retry count and a backoff time. Retry count sets the number of times the action is tried before being considered a fatal error. The backoff time is the time interval between retries.

Note:
Throwing a non-retryable exception on any retry iteration is treated as a fatal exception and the retries stop.

Below is a list of adapter objects and how retry behavior applies to each:

Beginning of the pool startup. If the backend system is down, and the user tries to enable the pool from Integration Server Administrator, the pool startup retries until the number of retries is exhausted or the retryable exception goes away. If the retries are exhausted, the pool is disabled.

Listeners. The retry process occurs during the listener startup or during listener execution. If the number of retries is exhausted, the listener is disabled. During execution, if the listener notification produces an AdapterConnectionException, the corresponding listener goes into the retry mode.

Adapter Services. Integration Server provides retry mechanism on services invoked from the triggers or flows invoked. An AdapterConnectionException caught from the adapter services execute method, is rethrown as an ISRuntimeException. Integration Server recognizes ISRuntimeException as a transient error and the retry cycle begins.

Polling Notifications. The polling notification retry works differently from the other components. If a transient error is detected during the scheduled execution, the connection is cleaned. The next scheduled interval acts as the retry back off time. However, there is no maximum number of attempts. The notification remains enabled and executes on its next scheduled interval.

The diagram shows the interactions with an adapter during an adapter service retry scenario.

The transient error condition is initially detected in the adapter service's execute method (step 2).

The adapter throwing the AdapterConnectionException resets the connection pool by destroying all the idle connections immediately and any active connections as they return to the connection pool (step 3).

When the connection pool is reset, the error is sent to the Integration Server service invocation logic as a retryable exception.

If so configured, and after an appropriate timeout period, the adapter service is again invoked (beginning at step 4).

Since the connection pool was emptied after the AdapterConnectionException, a new connection is created for the service invocation (steps 5 and 6).

If a transient error condition occurs, then another AdapterConnectionException must be thrown from either the createManagedConnectionObject method or initializeConnection method call. The pool is reset again (step 3) and the retry process begins.