Restart After Unplanned Shutdown
Software AG recommends that you always back up your database before performing any changes to the cluster, even if you do not plan to restore the backup data afterwards.
If only the primary node has shut down unexpectedly, you can restart it as a secondary node.
If all nodes have shut down unexpectedly, restart the primary node first and then the secondary node(s).
1. To restart the failed node as a secondary node:
a. Ensure that CLUSTER_URL is defined and CLUSTER_BOOTSTRAP is unset in the DBnnn.INI file.
To check the DBnnn.INI file:
adaini dbid=dbid show topic=nucparms item=CLUSTER_URL
adaini dbid=dbid show topic=nucparms item=CLUSTER_BOOTSTRAP
To check the environment variable setting:
set -u
echo $CLUSTER_BOOTSTRAP
echo $CLUSTER_URL
Software AG recommends setting CLUSTER_URL in the DBdbid.INI file and CLUSTER_BOOTSTRAP as an environment variable only if necessary.
b. Start the secondary node using the following command:
adastart dbid
2. If all nodes have shut down unexpectedly:
a. Identify the node with the highest sequence number (the primary node).
To identify the seqno, execute the following command on every node:
adanuc dbid=dbid noplog options=autorestart_only
In the resulting output, the GTID (global transaction ID) contains the cluster ID and sequence number in the following format: GTID = cluster-id:seqno.
Bootstrapping a node with a lower sequence number might result in issues when rejoining the remaining nodes to the cluster.
b. Ensure that safe_to_bootstrap: 1 in $ADADATADIR/dbxxx/grastate.dat.
To set safe_to_bootstrap: 1, you may execute the following command:
sed -i "/safe_to_bootstrap/s/0/1/" $ADADATADIR/dbdbid/grastate.dat
c. Set the CLUSTER_BOOTSTRAP environment variable for the node with the highest seqno with the following command:
export CLUSTER_BOOTSTRAP=
d. Start the node with the highest sequence number (primary node) with the following command:
adastart dbid
e. Remove the CLUSTER_BOOTSTRAP environment variable using the following command:
unset CLUSTER_BOOTSTRAP
f. For the secondary node(s), ensure that the CLUSTER_URL setting is valid in the DBdbid.INI file and CLUSTER_BOOTSTRAP is not set.
g. Start the secondary node(s) using the following command:
adastart dbid
Starting an aborted nucleus recovers the applied transactions and updates the seqno of the node.