BigMemory Max 4.3.1 | Product Documentation | Big Memory Max High-Availability Guide | Testing High-Availability Deployments | Terracotta Cluster Tests
 
Terracotta Cluster Tests
All tests in this section should be run after the Network Tests succeed.
Test Plan - Active L2 System Loss Tests - verify Mirror Takeover
The test plan for mirror takeover consists of the following tests:
TestID
Test
Setup
Steps
Expected Result
TAL1
Active L2 Loss - Kill
L2-A is active, L2-B is mirror. All systems are running and available to take traffic.
1. Run app
2. Kill -9 Terracotta PID on L2-A (Active)
L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL2
Active L2 Loss - clean shutdown
L2-A is active, L2-B is mirror. All systems are running and available to take traffic.
1. Run app
2. Run ~/bin/stop-tc-server.sh on L2-A (Active)
L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL3
Active L2 Loss - Power Down
L2-A is Active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Power down L2-A (Active)
L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL4
Active L2 Loss - Reboot
L2-A is Active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Reboot L2-A (Active)
L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
TAL5
Active L2 Loss - Pull Plug
L2-A is Active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Pull the power cable on L2-A (Active)
L2-B(mirror) becomes active. Takes the load. No drop in TPS on Failover.
Test Plan - Mirror L2 System Loss Tests
System loss tests confirms High Availability in the event of loss of a single system. This section outlines tests for testing failure of the Terracotta mirror server.
The test plan for testing Terracotta mirror Failures consist of the following tests:
TestID
Test
Setup
Steps
Expected Result
TPL1
Mirror L2 loss - kill
L2-A is active, L2-B is mirror. All systems are running and available to take traffic.
1. Run app
2. Kill -9 L2-B (mirror)
data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL2
Mirror L2 loss -clean
L2-A is active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Run ~/bin/stop-tc-server.sh on L2-B (mirror)
data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL3
Mirror L2 loss -power down
L2-A is active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Power down L2-B (mirror)
data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL4
Mirror L2 loss -reboot
L2-A is active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Reboot L2-B (mirror)
data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
TPL5
Mirror L2 loss -Pull Plug
L2-A is active, L2-B is mirror. All systems are running and available to take traffic
1. Run app
2. Pull plug on L2-B (mirror)
data directory needs to be cleaned up, then when L2-B is restarted, it re-synchs state from Active Server.
Test Plan - Failover/Failback Tests
This section outlines tests to confirm the cluster ability to fail-over to the mirror Terracotta server, and fail back.
The test plan for testing fail over and fail back consists of the following tests:
TestID
Test
Setup
Steps
Expected Result
TFO1
Failover/Failback
L2-A is active, L2-B is mirror. All systems are running and available to take traffic
1. Run application
2. Kill -9 (or run stop-tc-server) on L2-A (Active)
3. After L2-B takes over as Active, start-tc-server on L2-A. (L2-A is now mirror)
4. Kill -9 (or run stop-tc-server) on L2-B. (L2-A is now Active)
After first failover L2-A->L2-B, txns should continue. L2-A should come up cleanly in mirror mode when tc-server is run. When second failover occurs L2-B->L2-A, L2-A should process txns.
Test Plan - Loss of Switch Tests
Tip: This test can only be run on a redundant network
This section outlines testing the loss of a switch in a redundant network, and confirming that no interrupt of service occurs.
The test plan for testing failure of a single switch consists of the following tests:
TestID
Test
Setup
Steps
Expected Result
TSL1
Loss of 1 Switch
2 Switches in redundant configuration. L2-A is active, L2-B is mirror. All systems are running and available to take traffic.
1. Run application
2. Power down/pull plug on Switch
All traffic transparently moves to switch 2 with no interruptions
Test Plan - Loss of Network Connectivity
This section outlines testing the loss of network connectivity.
The test plan for testing failure of the network consists of the following tests:
TestID
Test
Setup
Steps
Expected Result
TNL1
Loss of NIC wiring (Active)
L2-A is active, L2-B is mirror. All systems are runnng and available to traffic
1. Run application
2. Remove Network Cable on L2-A
All traffic transparently moves to L2-B with no interruptions
TNL2
Loss of NIC wiring (mirror)
L2-A is active, L2-B is mirror. All systems are runnng and available to traffic
1. Run application
2. Remove Network Cable on L2-B
No user impact on cluster
Test Plan - Terracotta Cluster Failure
This section outlines the tests to confirm successful continued operations in the face Terracotta Cluster failures.
The test plan for testing Terracotta Cluster failures consists of the following tests:
TestID
Test
Setup
Steps
Expected Result
TF1
Process Failure Recovery
L2-A is active, L2-B is mirror. All systems are running and available to traffic
1. Run application
2. Bring down all L1s and L2s
3. Start L2s then L1s
Cluster should come up and begin taking txns again
TF2
Server Failure Recovery
L2-A is active, L2-B is mirror. All systems are running and available to traffic
1. Run application
2. Power down all machines
3. Start L2s and then L1s
Should be able to run application once all servers are up.
Client Failure Tests
This section outlines tests to confirm successful continued operations in the face of Terracotta client failures.
The test plan for testing Terracotta Client failures consists of the following tests:
TestID
Test
Setup
Steps
Expected Result
TCF1
L1 Failure -
L2-A is active, L2-B is mirror. 2 L1s L1-A and L1-B All systems are running and available to traffic
1. Run application
2. kill -9 L1-A.
L1-B should take all incoming traffic. Some timeouts may occur due to txns in process when L1 fails over.
Copyright © 2010 - 2019 | Software AG, Darmstadt, Germany and/or Software AG USA, Inc., Reston, VA, USA, and/or its subsidiaries and/or its affiliates and/or their licensors.