Terracotta Cluster Tests

BigMemory 4.4.0 | Product Documentation | BigMemory Max High-Availability Guide | Testing High-Availability Deployments | Terracotta Cluster Tests

Terracotta Cluster Tests

All tests in this section should be run after the Network Tests succeed.

Test Plan - Active L2 System Loss Tests - verify Mirror Takeover

The test plan for mirror takeover consists of the following tests:

TestID	Test	Setup	Steps	Expected Result
TAL1	Active L2 Loss - Kill	L2-A is active, L2-B is mirror. All systems are running and available to take traffic.	1. Run app 2. Kill -9 Terracotta PID on L2-A (Active)	L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL2	Active L2 Loss - clean shutdown	L2-A is active, L2-B is mirror. All systems are running and available to take traffic.	1. Run app 2. Run ~/bin/stop-tc-server.sh on L2-A (Active)	L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL3	Active L2 Loss - Power Down	L2-A is Active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Power down L2-A (Active)	L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL4	Active L2 Loss - Reboot	L2-A is Active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Reboot L2-A (Active)	L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL5	Active L2 Loss - Pull Plug	L2-A is Active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Pull the power cable on L2-A (Active)	L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.

Test Plan - Mirror L2 System Loss Tests

System loss tests confirms High Availability in the event of loss of a single system. This section outlines tests for testing failure of the Terracotta mirror server.

The test plan for testing Terracotta mirror Failures consist of the following tests:

TestID	Test	Setup	Steps	Expected Result
TPL1	Mirror L2 loss - kill	L2-A is active, L2-B is mirror. All systems are running and available to take traffic.	1. Run app 2. Kill -9 L2-B (mirror)	data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL2	Mirror L2 loss -clean	L2-A is active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Run ~/bin/stop-tc-server.sh on L2-B (mirror)	data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL3	Mirror L2 loss -power down	L2-A is active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Power down L2-B (mirror)	data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL4	Mirror L2 loss -reboot	L2-A is active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Reboot L2-B (mirror)	data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL5	Mirror L2 loss -Pull Plug	L2-A is active, L2-B is mirror. All systems are running and available to take traffic	1. Run app 2. Pull plug on L2-B (mirror)	data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.

Test Plan - Failover/Failback Tests

This section outlines tests to confirm the cluster ability to fail-over to the mirror Terracotta server, and fail back.

The test plan for testing fail over and fail back consists of the following tests:

TestID	Test	Setup	Steps	Expected Result
TFO1	Failover/Failback	L2-A is active, L2-B is mirror. All systems are running and available to take traffic	1. Run application 2. Kill -9 (or run stop-tc-server) on L2-A (Active) 3. After L2-B takes over as Active, start-tc-server on L2-A. (L2-A is now mirror) 4. Kill -9 (or run stop-tc-server) on L2-B. (L2-A is now Active)	After first failover L2-A->L2-B, txns should continue. L2-A should come up cleanly in mirror mode when tc-server is run. When second failover occurs L2-B->L2-A, L2-A should process txns.

Test Plan - Loss of Switch Tests

Tip:
This test can only be run on a redundant network

This section outlines testing the loss of a switch in a redundant network, and confirming that no interrupt of service occurs.

The test plan for testing failure of a single switch consists of the following tests:

TestID	Test	Setup	Steps	Expected Result
TSL1	Loss of 1 Switch	2 Switches in redundant configuration. L2-A is active, L2-B is mirror. All systems are running and available to take traffic.	1. Run application 2. Power down/pull plug on Switch	All traffic transparently moves to switch 2 with no interruptions

Test Plan - Loss of Network Connectivity

This section outlines testing the loss of network connectivity.

The test plan for testing failure of the network consists of the following tests:

TestID	Test	Setup	Steps	Expected Result
TNL1	Loss of NIC wiring (Active)	L2-A is active, L2-B is mirror. All systems are runnng and available to traffic	1. Run application 2. Remove Network Cable on L2-A	All traffic transparently moves to L2-B with no interruptions
TNL2	Loss of NIC wiring (mirror)	L2-A is active, L2-B is mirror. All systems are runnng and available to traffic	1. Run application 2. Remove Network Cable on L2-B	No user impact on cluster

Test Plan - Terracotta Cluster Failure

This section outlines the tests to confirm successful continued operations in the face Terracotta Cluster failures.

The test plan for testing Terracotta Cluster failures consists of the following tests:

TestID	Test	Setup	Steps	Expected Result
TF1	Process Failure Recovery	L2-A is active, L2-B is mirror. All systems are running and available to traffic	1. Run application 2. Bring down all L1s and L2s 3. Start L2s then L1s	Cluster should come up and begin taking txns again
TF2	Server Failure Recovery	L2-A is active, L2-B is mirror. All systems are running and available to traffic	1. Run application 2. Power down all machines 3. Start L2s and then L1s	Should be able to run application once all servers are up.

Client Failure Tests

This section outlines tests to confirm successful continued operations in the face of Terracotta client failures.

The test plan for testing Terracotta Client failures consists of the following tests:

TestID	Test	Setup	Steps	Expected Result
TCF1	L1 Failure -	L2-A is active, L2-B is mirror. 2 L1s L1-A and L1-B All systems are running and available to traffic	1. Run application 2. kill -9 L1-A.	L1-B should take all incoming traffic. Some timeouts may occur due to txns in process when L1 fails over.