Adding a ZooKeeper Role
Describes how to add a ZooKeeper role to a Data Fabric node by using the CLI.
About this task
This procedure adds one additional ZooKeeper to a previously installed cluster and uses rolling restarts to avoid cluster outages. If multiple additional ZooKeeper instances need to be added, repeat this procedure from start to finish for each new ZooKeeper.
While it is possible to use a modified version of this procedure to add multiple additional ZooKeepers at once, doing so involves more complexity and carries increased risk of creating a cluster outage if incorrect steps are followed.
Procedure
- On each previously installed node, identify the state of the ZooKeeper
service:
All previously installed ZooKeeper services must be running and in either the leader or follower state. Do not proceed further until all previously installed ZooKeeper services are running as expected./opt/mapr/initscripts/zookeeper qstatus
- Install the ZooKeeper package on the new node. For more information about packages, see Data Fabric Repositories and Packages.
- Copy the tokens and all other directories needed from a previously installed
node to the new node. You can find the list of files you need to copy in
Enabling Security on a Configured Cluster.IMPORTANTCopying files while logged on as the
root
user results in the files being unreadable. Make sure the copied files have identical ownership and permissions between the previously installed node and the new node. - On all nodes in the cluster, run configure.sh
with the following options:
-Z
: Specify all previously installed ZooKeeper nodes and the new node.-R
: Specify this option to use all previously set configurations.
To identify the list of current ZooKeeper nodes in the cluster, run/opt/mapr/server/configure.sh -Z <all-nodes-including-new-one> -R
maprcli node listzookeepers
. - Perform a rolling restart of Warden on all nodes. This step must be done
before the ZooKeeper service is started on the new node as it is possible
the new node becomes the leader, in which case, other cluster services would
be unaware of where to find the leader. Run the following command on all nodes to restart Warden:
systemctl restart mapr-warden
- Perform a rolling restart of the ZooKeeper services on the previously
installed and new nodes. Use the following command to restart the ZooKeeper
service:
For each previously installed ZooKeeper node, one by one, restart the ZooKeeper service and then wait until the state is shown as either leader or follower before proceeding to the next node. Do not proceed to the next node until the state is shown as expected.systemctl restart mapr-zookeeper
After all previously installed ZooKeeper services have been restarted, start the ZooKeeper service on the new node.
- Confirm that the state of the ZooKeeper service on all nodes shows as either leader or follower.
- Run
maprcli node listzookeepers
. The output should show all ZooKeeper nodes, including the node where the role was added.