Diagnosing and Repairing Problems
Diagnostic
The diagnostic command displays detailed status information for each node in the cluster including: online, activation, health, restart and state. Last changed configuration details are also displayed.
Syntax
diagnostic -connect-to <hostname[:port]>
Option | Description |
-connect-to <hostname[:port] | Terracotta server instance to which the Config Tool will connect and execute the command. |
Example of detailed status/diagnostic information for the cluster:
config-tool.sh diagnostic -connect-to host1:9410
Diagnostic result:
[Cluster]
- Nodes: 7 (node1@host1:9410, node2@host2:9412, node3@host3:9413, node4@host4:9414, node5@host5:9415, node6@host6:9416, node7@host7:9417)
- Nodes online: 7 (host1:9410, host2:9412, host3:9413, host4:9414, host5:9415, host6:9416, host7:9417)
- Nodes online, configured and activated: 7 (host1:9410, host2:9412, host3:9413, host4:9414, host5:9415, host6:9416, host7:9417)
- Nodes online, configured and in repair: 0
- Nodes online, new and being configured: 0
- Nodes pending restart: 7 (host1:9410, host2:9412, host3:9413, host4:9414, host5:9415, host6:9416, host7:9417)
- Configuration state: The cluster configuration is healthy and all nodes are online. No repair needed. New configuration changes are possible.
[node1@host1:9410]
- Node state: PASSIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
[node2@host2:9412]
- Node state: ACTIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by: jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
[node3@host3:9413]
- Node state: ACTIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by: jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
[node4@host4:9414]
- Node state: PASSIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by: jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
[node5@host5:9415]
- Node state: PASSIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by: jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
[node6@host6:9416]
- Node state: PASSIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by: jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
[node7@host7:9417]
- Node state: ACTIVE
- Node online, configured and activated: YES
- Node online, configured and in repair: NO
- Node online, new and being configured: NO
- Node restart required: YES
- Node configuration change in progress: NO
- Node can accept new changes: YES
- Node current configuration version: 3
- Node highest configuration version: 3
- Node last configuration change UUID: 7a37809c-e6c0-42f8-8c61-e2a47142b38f
- Node last configuration state: COMMITTED
- Node last configuration created at: 2021-02-18T14:57:54.063
- Node last configuration created from: COMPANY-PC
- Node last configuration created by: jdoe
- Node last configuration change details: set backup-dir=new-backup-dir
- Node last mutation at: 2021-02-18T14:57:54.333
- Node last mutation from: COMPANY-PC
- Node last mutation by: jdoe
In the above example, the diagnostic output contains the following Configuration state message near the top of the listing:
Configuration state: The cluster configuration is healthy and all nodes are online. No repair needed. New configuration changes are possible.
This specific message represents a perfectly healthy cluster. But if the cluster is not healthy, then the reason for the unhealthy state will be captured in the 'Configuration state' message block. Refer to the
Troubleshooting Guide for guidance on how to interpret these messages and for guidance on how to effect the necessary repairs to the cluster.
Repair
The repair command is used to repair the cluster health by fixing cluster configuration inconsistency issues on one or more nodes targeting single node repairs. Repairs only work on activated nodes - not nodes that are running in diagnostic mode. Running this command on a healthy cluster has no effect.
Important:
Please contact Software AG support before running this command.
Syntax:
repair -connect-to <hostname[:port]>
Option | Description |
-connect-to <hostname[:port] | Terracotta server instance to which the Config Tool will connect and execute the repair. |
Important:
In the event that problems are reported by the Config Tool when executing the
repair command, note the error message and refer to the
Troubleshooting Guide for guidance.
Example of repairing a node:
config-tool.sh repair -connect-to host1:9410
Log
Retrieve details about all changes made to a node in the cluster.
Syntax:
log -connect-to <hostname[:port]>
Option | Description |
-connect-to <hostname[:port] | Terracotta server instance to which the Config Tool will connect and execute the command to retrieve the node's logged changes. |
Example of a query to a node (node7) for all configuration changes:
config-tool.sh log -connect-to host7:9417
1 2021-02-18T14:21:46.030 c7ac4b3c-bb5e-4481-a259-bace0ce4abbb c3745cb968cd289ddabab70c9634e749823289cd COMMITTED | jdoe@COMPANY-PC - Activating cluster: MyCluster
2 2021-02-18T14:57:11.298 64efd21b-5806-4d38-b3d2-a68f9e045378 1f050acacbdb1732e76bce60a9494bdde5dd80a5 COMMITTED | jdoe@COMPANY-PC - unset backup-dir
3 2021-02-18T14:57:54.063 7a37809c-e6c0-42f8-8c61-e2a47142b38f e3661632e53ae15245ad8632c0b0dacd9faf1449 COMMITTED | jdoe@COMPANY-PC - set backup-dir=new-backup-dir