Recommended Settings for Planned Shutdown

Explains the modalities of a planned shutdown.

The HPE Ezmeral Data Fabric stack improves the latencies for planned shutdowns by implementing a fast failover mechanism where different services respond to the intimation of a failure.

Notifying CLDB to Allow Fast Failover

When planning to shutdown a node, notifying CLDB of an impending shutdown allows CLDB to update the replication chain such that primary and intermediate containers, if any, are not on the node and re-assign VIPs on the node when the node actually goes down. This, in turn, allows clients to continue activities on available nodes.

MapR (v5.1) includes an argument, node failover, to the maprcli command that notifies CLDB of impending node shutdown so that CLDB can ensure that the specified node does not have any primary containers and intermediate containers (in a cascaded chain), and VIPs are re-assigned.

Shutting Down a Node

To notify CLDB of a planned shutdown of a node:

  1. Enable the fast failover behavior.

    Refer to Enabling Fast Failover for more information.

  2. Reset the value of tcp_syn_retries parameter.

    Refer to Tuning TCP for more information.

  3. (Optional) Get the hostname of the node to put in maintenance mode by running the following command: 
    /opt/mapr/bin/maprcli node list -columns hostname
  4. Run the failover command for that node.
    For example: 
    /opt/mapr/bin/maprcli node failover -nodes <node-hostname>
    Wait for few minutes (to allow containers to failover) before proceeding to the next step.
  5. Stop warden on that node by running the following command: 
    service mapr-warden stop
  6. Notify HPE Ezmeral Data Fabric that the node is in maintenance mode and when the maintenance task is complete, remove the node from maintenance mode.

    See Performing Maintenance on a Node for the commands to run to put and take a node out of maintenance mode.

WARNING
Shut down only one node at a time. Do not take down multiple nodes for maintenance at the same time.