Adding a ZooKeeper Role

Describes how to add a ZooKeeper role to a Data Fabric node by using the CLI.

About this task

This procedure adds one additional ZooKeeper to a previously installed cluster and uses rolling restarts to avoid cluster outages. If multiple additional ZooKeeper instances need to be added, repeat this procedure from start to finish for each new ZooKeeper.

While it is possible to use a modified version of this procedure to add multiple additional ZooKeepers at once, doing so involves more complexity and carries increased risk of creating a cluster outage if incorrect steps are followed.

Procedure

  1. On each previously installed node, identify the state of the ZooKeeper service:
    /opt/mapr/initscripts/zookeeper qstatus
    All previously installed ZooKeeper services must be running and in either the leader or follower state. Do not proceed further until all previously installed ZooKeeper services are running as expected.
  2. Install the ZooKeeper package on the new node. For more information about packages, see Data Fabric Repositories and Packages.
  3. Copy the tokens and all other directories needed from a previously installed node to the new node. You can find the list of files you need to copy in Enabling Security.
    IMPORTANT
    Copying files while logged on as the root user results in the files being unreadable. Make sure the copied files have identical ownership and permissions between the previously installed node and the new node.
  4. On all nodes in the cluster, run configure.sh with the following options:
    • -Z: Specify all previously installed ZooKeeper nodes and the new node.
    • -R: Specify this option to use all previously set configurations.
    For example:
    /opt/mapr/server/configure.sh -Z <all-nodes-including-new-one> -R
    To identify the list of current ZooKeeper nodes in the cluster, run maprcli node listzookeepers.
  5. Perform a rolling restart of Warden on all nodes. This step must be done before the ZooKeeper service is started on the new node as it is possible the new node becomes the leader, in which case, other cluster services would be unaware of where to find the leader.
    Run the following command on all nodes to restart Warden:
    systemctl restart mapr-warden
  6. Perform a rolling restart of the ZooKeeper services on the previously installed and new nodes. Use the following command to restart the ZooKeeper service:
    systemctl restart mapr-zookeeper
    For each previously installed ZooKeeper node, one by one, restart the ZooKeeper service and then wait until the state is shown as either leader or follower before proceeding to the next node. Do not proceed to the next node until the state is shown as expected.

    After all previously installed ZooKeeper services have been restarted, start the ZooKeeper service on the new node.

  7. Confirm that the state of the ZooKeeper service on all nodes shows as either leader or follower.
  8. Run maprcli node listzookeepers. The output should show all ZooKeeper nodes, including the node where the role was added.