Event Types and Definitions

This section describes the types of events that can be found in logs or viewed in the TMC.

Cause: A full garbage collection (GC) longer than the configured threshold has occurred.

Action: Reduce cache memory footprint in L1 (Terracotta client). Investigate issues with application logic and garbage creation.

Notes: The default critical threshold is 8 seconds, but it can be reconfigured in tc.properties using longgc.threshhold. For information about setting tc.properties, see the Terracotta Configuration Parameters.

Occurrence of this event could help diagnose certain failures. For details, see "Configuring the HealthChecker Properties" in the BigMemory Max High-Availability Guide.

Cause: Periodic distributed garbage collection (DGC), which was explicitly enabled in the configuration, has started a cleanup cycle.

Action: If periodic DGC is unneeded, disable it to improve overall cluster performance.

Notes: Periodic DGC, which is disabled by default, is mostly useful in the absence of automatic handling of distributed garbage.

Cause: Periodic DGC, which was explicitly enabled in the configuration, ended a cleanup cycle.

Action: If periodic DGC is unneeded, disable it to improve overall cluster performance.

Notes: Event message reads "DGC[ {0} ] finished. Begin Count : {1} Collected : {2} Time Taken : {3} ms Live Objects : {4}".

Cause: Periodic DGC, wPeriodichich was explicitly enabled in the configuration, has been cancelled due to an interruption (for example, by a failover operation).

Action: If periodic DGC is unneeded, disable it to improve overall cluster performance.

Notes: Periodic DGC, which is disabled by default, is mostly useful in the absence of automatic handling of distributed garbage.

Cause: L2 (Terracotta server) is starting up as ACTIVE with existing data, triggering inline distributed garbage collection (DGC).

Action: No action necessary.

Notes: Only seen when a server starts up as ACTIVE upon a recovery, using Fast Restartability.

Cause: Inline DGC operation completed.

Action: No action necessary.

Notes: Event message reads "Inline DGC [ {0} ] reference cleanup finished. Begin Count : {1} Collected : {2} Time Taken : {3} ms Live Objects : {4}".

Cause: Inline DGC operation interrupted.

Action: Investigate any unusual cluster behavior or other events.

Notes: Possibly occurs during failover, but other events should indicate real cause.

Cause: Specified node has joined the cluster.

Action: No action necessary.

Cause: Specified node has left the cluster.

Action: Check why the node has left (for example: long GC, network issues, or issues with local node resources).

Cause: L2 changing state (for example, from INITIALIZING to ACTIVE).

Action: Check to see that the state change is expected.

Notes: Event message reads "Moved to {0}", where {0} is the new state.

Cause: L1 is unsuccessfully trying to reconnect to cluster, but it has already been expelled.

Action: If the L1 does not go into a rejoin operation, it must be restarted manually.

Notes: Event message reads "An {0} client {1} tried to connect to {2} server. Connection refused!!"

Cause: Active server left the cluster.

Action: Check why the active L2 has left.

Cause: Mirror server left the cluster.

Action: Check why the mirror L2 has left.

Cause: One L2 is trying to cause another L2 to restart ("zap").

Action: Investigate a possible "split brain" situation (a mirror L2 behaves as the ACTIVE) if the zapped L2 does not obey the restart order.

Notes: A "zap" operation happens only within a mirror group. Event message reads "SPLIT BRAIN, {0} and {1} are ACTIVE", where {0} and {1} are the two servers vying for the ACTIVE role.

Cause: The L2 is accepting the order to restart ("zap" order).

Action: Check the state of the zapped L2 to ensure that it restarts as a mirror, or manually restart it.

Notes: A "zap" order is issued only within a mirror group. Event message reads "{0} has more clients. Exiting!!", where {0} is the L2 that becomes the ACTIVE.

Cause: A mirror L2 is trying to join with data in place.

Action: If the mirror does not automatically restart and wipe its data, its data may need to be manually wiped and before it is restarted.

Notes: Restarted mirror L2s must wipe their data to resync with the active L2. This is normally an automatic operation that should not require action. Event message reads "Started with dirty database. Exiting!! Restart {0}", where {0} is the the mirror that is automatically restarting.

Cause: Cluster configuration was reloaded.

Action: No action necessary.

Cause: Automatic evictions for optimizing Terracotta Server Array operations.

Action: No action necessary.

Notes: Event message reads "DCV2 Eviction - Time taken (msecs)={0}, Number of entries evicted={1}, Number of segments over threshold={2}, Total Overshoot={3}".

Cause: System clocks are not aligned.

Action: Synchronize system clocks.

Notes: The default tolerance is 30 seconds, but it can be reconfigured in tc.properties using time.sync.threshold. For information about setting tc.properties, see Terracotta Configuration Parameters.

Note that overly large tolerance can introduce unpredictable errors and behaviors.

Cause: L2 entered throttled mode, which could be a temporary condition (e.g., caused by bulk-loading) or could indicate insufficient allocation of memory.

Action: See Managing Near-Memory-Full Conditions.

Notes: After emitting this, L2 can emit resource.capacityrestored (return to normal mode) or resource.fullcapacity (move to restricted mode), based on resource availability. Event message reads "{0} is nearing capacity limit, performance may be degraded - {1}% usage", where {0} is the L2 identification and {1} is the % usage of the memory resources allocated to that L2.

Cause: L2 entered restricted mode, which could be a temporary condition (e.g., caused by bulk-loading) or could indicate insufficient allocation of memory.

Action: See Managing Near-Memory-Full Conditions.

Notes: After emitting this, L2 can emit resource.capacityrestored (return to normal mode), based on resource availability. Event message reads "{0} is at over capacity limit, no further additive operations will be accepted - {1}% usage", where {0} is the L2 identification and {1} is the % usage of the memory resources allocated to that L2.

Cause: L2 returned to normal from throttled or restricted mode.

Action: No action necessary.

Notes: Event message reads "{0} capacity has been restored, performance has returned to normal - {1}% usage", where {0} is the L2 identification and {1} is the % usage of the memory resources allocated to that L2.