Setting up API Portal HA setup
This procedure describes in detail the setting up of the HA setup for API Portal.
Prerequisites:
Ensure to follow the below to set up HA and update the fixes.
Install
API Portal but do not start any runnables.
Note: To install fix on a HA setup:
Use UpdateManager to install the fixes on the nodes where apiportalbundle is configured.
Post fix installation, connect to the cluster environment using ACC console and ensure all runnables are in STARTED state.
To configure
API Portal HA set up
1. Add worker nodes to ACC.
a. Start ACC.
b. Execute the add node command for each of the worker nodes. The basic syntax of the add node command is as follows:
add node logicalNodeName ipAddressOrHostname
@agentPort] agentUsername agentPassword
Replace logicalNodeName with the logical node name you want to assign to that node and with which you will later refer to it in the ACC commands.
2. Create a 3-node environment.
a. On machine1, create a nodelist.pt file in the folder C:\SoftwareAG\API_Portal\server, which contains the following lines:
add node n1 machine1 @18010 Clous g3h31m
add node n2 machine2 @18010 Clous g3h31m
add node n3 machine3 @18010 Clous g3h31m
set current node n1
b. Replace machine1, machine2, and machine3 with the names or IP addresses of your machines.
c. Run the following command:
C:\SoftwareAG\API_Portal\server>acc\acc.bat -n
C:\SoftwareAG\API_Portal\server\nodelist.pt –c
C:\SoftwareAG\API_Portal\server\generated.apptypes.cfg
This creates an ensemble between the instances in the cluster.
d. To view the 3-node cluster in ACC, run the command:
ACC+ n1>list nodes
the 3-node cluster has all nodes listed listening on port 18010 using REST services as follows:
n1 : machine1 (18010) OK
n2 : machine2 (18010) OK
n3 : machine3 (18010) OK
3. Cleanup the unnecessary runnables by running the following commands, in ACC, to deconfigure the runnables for all the 3 nodes:
ACC+ n1>on n1 deconfigure zoo_s
ACC+ n1>on n1 deconfigure cloudsearch_s
ACC+ n1>on n1 deconfigure apiportalbundle_s
ACC+ n1>on n2 deconfigure zoo_s
ACC+ n1>on n2 deconfigure postgres_s
ACC+ n1>on n3 deconfigure zoo_s
ACC+ n1>on n3 deconfigure postgres_s
4. Create a zookeeper cluster in ACC by running the following commands:
ACC+ n1>on n1 add zk
ACC+ n1>on n2 add zk
ACC+ n1>on n3 add zk
ACC+ n1>commit zk changes
You can view the following configuration using the ACC command after you start the zoo runnables:
ACC+ n1>list zk instances
3 Zookeeper instances:
Node InstID MyID State Cl Port Port A Port B Type
n1 zoo0 1 STARTED 14281 14285 14290 Master
n2 zoo0 2 STARTED 14281 14285 14290 Master
n3 zoo0 3 STARTED 14281 14285 14290 Master
5. Reconfigure the three elasticsearch runnables to form a cluster through ACC by running the following commands:
ACC+ n1>on n1 reconfigure elastic_s
+ELASTICSEARCH.node.name = machine1
+ELASTICSEARCH.cluster.name=apiportal
ELASTICSEARCH.discovery.zen.ping.unicast.hosts="machine2:esTCPport,
machine3:esTCPport"
+ELASTICSEARCH.discovery.zen.minimum_master_nodes=2 -zookeeper.connect.string
+ELASTICSEARCH.index.number_of_replicas=1
-ELASTICSEARCH.sonian.elasticsearch.zookeeper.client.host
ACC+ n1>on n2 reconfigure elastic_s
+ELASTICSEARCH.node.name = machine2
+ELASTICSEARCH.cluster.name=apiportal
ELASTICSEARCH.discovery.zen.ping.unicast.hosts="machine1:esTCPport,
machine3:esTCPport"
+ELASTICSEARCH.discovery.zen.minimum_master_nodes=2 -zookeeper.connect.string
+ELASTICSEARCH.index.number_of_replicas=1
-ELASTICSEARCH.sonian.elasticsearch.zookeeper.client.host
ACC+ n1>on n3 reconfigure elastic_s
+ELASTICSEARCH.node.name = machine3
+ELASTICSEARCH.cluster.name=apiportal
ELASTICSEARCH.discovery.zen.ping.unicast.hosts="machine1:esTCPport,
machine2:esTCPport"
+ELASTICSEARCH.discovery.zen.minimum_master_nodes=2 -zookeeper.connect.string
+ELASTICSEARCH.index.number_of_replicas=1
-ELASTICSEARCH.sonian.elasticsearch.zookeeper.client.host
Note: In the three commands above, replace machine1, machine2, and machine3 with the names or IP addresses of your machine s and esTCPport with elasticsearch TCP Port.
6. To validate the elasticsearch cluster, execute the below command:
validate elasticsearch cluster
This displays the following message:
Found 3 Elasticsearch instances in one
cluster across all currently registered nodes.
There were no errors.
7. To set the same user name and password for the ElasticSearch runnable in all three nodes, execute the below command:
For example,
on n1 reconfigure elastic_s +ELASTICSEARCH.aris.api.user.name="portal"
+ELASTICSEARCH.aris.api.user.password="manager"
8. Reconfigure kibana runnable on all three nodes as follows:
on n1 reconfigure kibana_s -zookeeper.connect.string
on n2 reconfigure kibana_s -zookeeper.connect.string
on n3 reconfigure kibana_s -zookeeper.connect.string
9. Reconfigure the PostgreSQL database on n1, so that it knows about all zookeeper cluster members and accepts connections from all locations, by running the following command:
on n1 reconfigure postgres_s -zookeeper.connect.string
+postgresql.listen_addresses = "'*'"
The only kind of scaling that is possible with PostgreSQL at the moment is that of scaling across tenants. Currently, the data of a single tenant always resides in a single database instance, so the load created by a single tenant on the database instance needs to be handled by that instance. At the same time, a tenant's database instance is a single point of failure for that tenant; if the tenant's DB goes offline, the tenant becomes unusable until the DB is available again. In particular for production use on mission critical systems, when high availability is of interest, this approach is not an ideal solution. Since
API Portal does not support a highly available configuration using our PostgreSQL runnable, you have to use an external DBMS like Oracle or MS SQL, which offer mechanisms for clustering and high availability. For details on configuring external databases, see
Configuring
API Portal with External Databases.
10. Define two cloudsearch instances, on the nodes n2 and n3, where each one belongs to a different data center:
on n2 reconfigure cloudsearch_s -zookeeper.connect.string
+zookeeper.application.instance.datacenter = n2
on n3 reconfigure cloudsearch_s -zookeeper.connect.string
+zookeeper.application.instance.datacenter = n3
11. Reconfigure the apiportalbundle runnable on the nodes n2 and n3 as follows:
on n2 reconfigure apiportalbundle_s -zookeeper.connect.string
on n3 reconfigure apiportalbundle_s -zookeeper.connect.string
12. Reconfigure the loadbalancer on n1 to point to all three zookeeper cluster members as follows:
on n1 reconfigure loadbalancer_s -zookeeper.connect.string
on n2 reconfigure loadbalancer_s -zookeeper.connect.string
on n3 reconfigure loadbalancer_s -zookeeper.connect.string
API Portal can be accessed through multiple hostnames. If one of the loadbalancer fails, the application can be accessed using the other available loadbalancers. You can ignore this step if external load balancer is not required. If you have an external load balancer Software AG recommends you to place a highly available loadbalancer (HA LB) in front of the loadbalancer runnables. To do this the following has to be added to the loadbalancer runnable configuration:
HTTPD.servername to specify the hostname or IP address of the HA loadbalancer
HTTPD.zookeeper.application.instance.port, the port on which the HA loadbalancer receives the http/https requests
zookeeper.application.instance.scheme which specifies the scheme http/https
For example:
on n1 reconfigure loadbalancer_s HTTPD.servername=HTTPD.servername
HTTPD.zookeeper.application.instance.port=PORT
zookeeper.application.instance.scheme=SCHEME
13. Change the startup order of the runnables by running the following commands:
ACC+ n1>on n1 set runnable.order = "zoo0 < (elastic_s, kibana_s, postgres_s)
< loadbalancer_s"
ACC+ n1>on n2 set runnable.order = "zoo0 < (elastic_s, kibana_s)
< cloudsearch_s < apiportalbundle_s < loadbalancer_s"
ACC+ n1>on n3 set runnable.order = "zoo0 < (elastic_s, kibana_s)
< cloudsearch_s < apiportalbundle_s < loadbalancer_s"
14. Create a file startupScript.bat under C:\SoftwareAG\API_Portal\server. Copy the following content into the file.
#
# start Zookeeper Ensemble
#
on n1 start zoo0
on n2 start zoo0
on n3 start zoo0
on n1 wait for STARTED of zoo0
on n2 wait for STARTED of zoo0
on n3 wait for STARTED of zoo0
#
# start ElasticSearch Cluster
#
on n1 start elastic_s
on n2 start elastic_s
on n3 start elastic_s
on n1 wait for STARTED of elastic_s
on n2 wait for STARTED of elastic_s
on n3 wait for STARTED of elastic_s
#
# start Kibana
#
on n1 start kibana_s
on n2 start kibana_s
on n3 start kibana_s
on n1 wait for STARTED of kibana_s
on n2 wait for STARTED of kibana_s
on n3 wait for STARTED of kibana_s
#
# start PostgreSQL Database
#
on n1 start postgres_s
on n1 wait for STARTED of postgres_s
#
# start CloudSearch
#
on n2 start cloudsearch_s
on n3 start cloudsearch_s
on n2 wait for STARTED of cloudsearch_s
on n3 wait for STARTED of cloudsearch_s
#
# start API Portal Bundle
#
on n2 start apiportalbundle_s
on n3 start apiportalbundle_s
on n2 wait for STARTED of apiportalbundle_s
on n3 wait for STARTED of apiportalbundle_s
#
# finally, start loadbalancer
#
on n1 start loadbalancer_s
on n2 start loadbalancer_s
on n3 start loadbalancer_s
on n1 wait for STARTED of loadbalancer_s
on n2 wait for STARTED of loadbalancer_s
on n3 wait for STARTED of loadbalancer_s
15. Execute the script by running the following command from the command prompt:
C:\SoftwareAG\API_Portal\server>acc\acc.sh |bat
-n C:\SoftwareAG\API_Portal\server\nodelist.pt -c
C:\SoftwareAG\API_Portal\server\generated.apptypes.cfg -f
C:\SoftwareAG\API_Portal\server\startupScript.bat
16. Ensure the HA setup is successfully running.
In the default installation, access the User Management Component (UMC) at http://machine name/umc, the ARIS Document Storage (ADS) at http://machine name/ads, the collaboration component at http://machine name/collaboration, and the API Portal at http:/machine name.
You can see that the elasticsearch cluster consists of three nodes, that the cluster name is apiportal and the master node is indicated by a solid star. Each index is split in 3 parts, called shards, which are replicated once and distributed over the three available nodes. If one node goes offline, the system can still fill up the complete index making it a fail-over system.