Setting up API Portal HA setup
This procedure describes in detail the setting up of the HA setup for API Portal.
Prerequisites:
Install API Portal but do not start any runnables.
To configure API Portal HA set up
1. Add worker nodes to ACC.
a. Start ACC.
b. Enter an add node command for each of the worker nodes. The basic syntax of the add node command is as follows:
add node <logicalNodeName> <ipAddressOrHostname>
[:<agentPort>] <agentUsername> <agentPassword>
Replace <logicalNodeName> with the logical node name you want to assign to that node and with which you will later refer to it in the ACC commands.
2. Create a 3-node environment.
a. On machine1, create a nodelist.pt file in the folder C:\SoftwareAG\API_Portal\server, which contains the following lines:
add node n1 machine1 @18008 Clous g3h31m
add node n2 machine2 @18008 Clous g3h31m
add node n3 machine3 @18008 Clous g3h31m
set current node n1
b. Replace machine1, machine2, and machine3 with the names or IP addresses of your machines.
c. Run the following command:
C:\SoftwareAG\API_Portal\server>acc\acc.bat -n
C:\SoftwareAG\API_Portal\server\nodelist.pt –c
C:\SoftwareAG\API_Portal\server\generated.apptypes.cfg
This creates an ensemble between the instances in the cluster.
d. To view the 3-node cluster in ACC, run the command:
ACC+ n1>list nodes
the 3-node cluster has all nodes listed listening on port 18008 using REST services as follows:
n1 : machine1 (18008) OK
n2 : machine2 (18008) OK
n3 : machine3 (18008) OK
3. Cleanup the unnecessary runnables by running the following commands, in ACC, to deconfigure the runnables for all the 3 nodes:
ACC+ n1>on n1 deconfigure zoo_s
ACC+ n1>on n1 deconfigure cloudsearch_s
ACC+ n1>on n1 deconfigure apiportalbundle_s
ACC+ n1>on n2 deconfigure zoo_s
ACC+ n1>on n2 deconfigure postgres_s
ACC+ n1>on n3 deconfigure zoo_s
ACC+ n1>on n3 deconfigure postgres_s
4. Create a zookeeper cluster in ACC by running the following commands:
ACC+ n1>on n1 add zk
ACC+ n1>on n2 add zk
ACC+ n1>on n3 add zk
ACC+ n1>commit zk changes
You can view the following configuration using the ACC command after you start the zoo runnables:
ACC+ n1>list zk instances
3 Zookeeper instances:
Node InstID MyID State Cl Port Port A Port B Type
n1 zoo0 1 STARTED 14281 14285 14290 Master
n2 zoo0 2 STARTED 14281 14285 14290 Master
n3 zoo0 3 STARTED 14281 14285 14290 Master
5. Reconfigure the three elasticsearch runnables to form a cluster through ACC by running the following commands:
ACC+ n1>on n1 reconfigure elastic_s
+ELASTICSEARCH.node.name = <machine1>
+ELASTICSEARCH.cluster.name=apiportal
ELASTICSEARCH.discovery.zen.ping.unicast.hosts="machine2:<esTCPport>,
machine3:<esTCPport>"
+ELASTICSEARCH.discovery.zen.minimum_master_nodes=2 -zookeeper.connect.string
+ELASTICSEARCH.index.number_of_replicas=1
-ELASTICSEARCH.sonian.elasticsearch.zookeeper.client.host
ACC+ n1>on n2 reconfigure elastic_s
+ELASTICSEARCH.node.name = <machine2>
+ELASTICSEARCH.cluster.name=apiportal
ELASTICSEARCH.discovery.zen.ping.unicast.hosts="machine1:<esTCPport>,
machine3:<esTCPport>"
+ELASTICSEARCH.discovery.zen.minimum_master_nodes=2 -zookeeper.connect.string
+ELASTICSEARCH.index.number_of_replicas=1
-ELASTICSEARCH.sonian.elasticsearch.zookeeper.client.host
ACC+ n1>on n3 reconfigure elastic_s
+ELASTICSEARCH.node.name = <machine3>
+ELASTICSEARCH.cluster.name=apiportal
ELASTICSEARCH.discovery.zen.ping.unicast.hosts="machine1:<esTCPport>,
machine2:<esTCPport>"
+ELASTICSEARCH.discovery.zen.minimum_master_nodes=2 -zookeeper.connect.string
+ELASTICSEARCH.index.number_of_replicas=1
-ELASTICSEARCH.sonian.elasticsearch.zookeeper.client.host
Note: In the three commands above, replace machine1, machine2, and machine3 with the names or IP addresses of your machine s and esTCPport with elasticsearch TCP Port.
6. To validate the elasticsearch cluster, execute the below command:
validate elasticsearch cluster
This displays the message: Found 3 Elasticsearch instances in one cluster across all currently registered nodes. There were no errors.
7. Reconfigure kibana runnable on the nodes n2 and n3 as follows:
on n1 reconfigure kibana_s -zookeeper.connect.string
on n2 reconfigure kibana_s -zookeeper.connect.string
on n3 reconfigure kibana_s -zookeeper.connect.string
8. Reconfigure the PostgreSQL database on n1, so that it knows about all zookeeper cluster members and accepts connections from all locations, by running the following command:
on n1 reconfigure postgres_s -zookeeper.connect.string
+postgresql.listen_addresses = "'*'"
The only kind of scaling that is possible with PostgreSQL at the moment is that of scaling across tenants. Currently, the data of a single tenant always resides in a single database instance, so the load created by a single tenant on the database instance needs to be handled by that instance. At the same time, a tenant's database instance is a single point of failure for that tenant; if the tenant's DB goes offline, the tenant becomes unusable until the DB is available again. In particular for production use on mission critical systems, when high availability is of interest, this approach is not an ideal solution. Since
API Portal does not support a highly available configuration using our PostgreSQL runnable, you have to use an external DBMS like Oracle or MS SQL, which offer mechanisms for clustering and high availability. For details on configuring external databases, see
Configuring
API Portal with External Databases.
9. Define two cloudsearch instances, on the nodes n2 and n3, where each one belongs to a different data center:
on n2 reconfigure cloudsearch_s -zookeeper.connect.string
+zookeeper.application.instance.datacenter = n2
on n3 reconfigure cloudsearch_s -zookeeper.connect.string
+zookeeper.application.instance.datacenter = n3
10. Reconfigure the apiportalbundle runnable on the nodes n2 and n3 as follows:
on n2 reconfigure apiportalbundle_s -zookeeper.connect.string
on n3 reconfigure apiportalbundle_s -zookeeper.connect.string
11. Reconfigure the loadbalancer on n1 to point to all three zookeeper cluster members as follows:
on n1 reconfigure loadbalancer_s -zookeeper.connect.string
on n2 reconfigure loadbalancer_s -zookeeper.connect.string
on n3 reconfigure loadbalancer_s -zookeeper.connect.string
API Portal can be accessed through multiple hostnames. If one of the loadbalancer fails, the application can be accessed using the other available loadbalancers. You can ignore this step if external load balancer is not required. If you have an external load balancer Software AG recommends you to place a highly available loadbalancer (HA LB) in front of the loadbalancer runnables. To do this the following has to be added to the loadbalancer runnable configuration:
HTTPD.servername to specify the hostname or IP address of the HA loadbalancer
HTTPD.zookeeper.application.instance.port, the port on which the HA loadbalancer receives the http/https requests
zookeeper.application.instance.scheme which specifies the scheme http/https
For example:
12. Change the startup order of the runnables by running the following commands:
ACC+ n1>on n1 set runnable.order = "zoo0 < (elastic_s, kibana_s, postgres_s)
< loadbalancer_s"
ACC+ n1>on n2 set runnable.order = "zoo0 < (elastic_s, kibana_s)
< cloudsearch_s < apiportalbundle_s < loadbalancer_s"
ACC+ n1>on n3 set runnable.order = "zoo0 < (elastic_s, kibana_s)
< cloudsearch_s < apiportalbundle_s < loadbalancer_s"
13. Create a file startupScript.bat under C:\SoftwareAG\API_Portal\server. Copy the following content into the file.
#
# start Zookeeper Ensemble
#
on n1 start zoo0
on n2 start zoo0
on n3 start zoo0
on n1 wait for STARTED of zoo0
on n2 wait for STARTED of zoo0
on n3 wait for STARTED of zoo0
#
# start ElasticSearch Cluster
#
on n1 start elastic_s
on n2 start elastic_s
on n3 start elastic_s
on n1 wait for STARTED of elastic_s
on n2 wait for STARTED of elastic_s
on n3 wait for STARTED of elastic_s
#
# start Kibana
#
on n1 start kibana_s
on n2 start kibana_s
on n3 start kibana_s
on n1 wait for STARTED of kibana_s
on n2 wait for STARTED of kibana_s
on n3 wait for STARTED of kibana_s
#
# start PostgreSQL Database
#
on n1 start postgres_s
on n1 wait for STARTED of postgres_s
#
# start CloudSearch
#
on n2 start cloudsearch_s
on n3 start cloudsearch_s
on n2 wait for STARTED of cloudsearch_s
on n3 wait for STARTED of cloudsearch_s
#
# start API Portal Bundle
#
on n2 start apiportalbundle_s
on n3 start apiportalbundle_s
on n2 wait for STARTED of apiportalbundle_s
on n3 wait for STARTED of apiportalbundle_s
#
# finally, start loadbalancer
#
on n1 start loadbalancer_s
on n2 start loadbalancer_s
on n3 start loadbalancer_s
on n1 wait for STARTED of loadbalancer_s
on n2 wait for STARTED of loadbalancer_s
on n3 wait for STARTED of loadbalancer_s
14. Execute the script by running the following command from the command prompt:
C:\SoftwareAG\API_Portal\server>acc\acc.bat –n
C:\SoftwareAG\API_Portal\server\nodelist.pt –c
C:\SoftwareAG\API_Portal\server\generated.apptypes.cfg -f
C:\SoftwareAG\API_Portal\server\startupScript.bat
15. Ensure the HA setup is successfully running.
In the default installation, access the User Management Component (UMC) at http://<machine name>/umc, the ARIS Document Storage (ADS) at http://<machine name>/ads, the collaboration component at http://<machine name>/collaboration, and the API Portal at http:/<machine name>.
You can see that the elasticsearch cluster consists of three nodes, that the cluster name is apiportal and the master node is indicated by a solid star. Each index is split in 3 parts, called shards, which are replicated once and distributed over the three available nodes. If one node goes offline, the system can still fill up the complete index making it a fail-over system.