System Maintenance

This topic describes the preparation and post-procedure tasks to perform maintenance, such as OS patches and upgrades, on hosts that are part of an HPE Ezmeral Runtime Enterprise deployment.

Use the information in this topic when you perform maintenance, such as OS patches and upgrades, on hybrid or on-premises hosts that are part of an HPE Ezmeral Runtime Enterprise deployment.

To update the Kubernetes version on a host, see Upgrading Kubernetes.

HPE Ezmeral Runtime Enterprise automatically drains the nodes during platform software upgrades and when upgrading Kubernetes to a newer version. In most cases, a best practice is to drain a Kubernetes node before performing system maintenance tasks such as OS kernel updates or hardware repairs. If you choose to drain a Kubernetes node, be aware that the HPE Ezmeral Runtime Enterprise software uses DaemonSet, so you must use the --ignore-daemonset option of the kubectl drain command. If your deployment uses custom DaemonSets or PodDisruptionBudgets, there might be other considerations when draining nodes. If you need help to evaluate your deployment and perform a system maintenance task, contact Hewlett Packard Enterprise Technical Support.

Preventing Unintended Updates of Kubernetes Packages

The yum update command, by default, attempts to update packages from all enabled repositories, including the repository that manages the kubeadm, kubelet, and kubecl packages. However, updating that repository using yum is not the correct upgrade procedure, and it can result in the installation of a package version that is not compatible with the current HPE Ezmeral Runtime Enterprise deployment, which leads to failures that result in applications not running correctly. In addition, the installed package version no longer matches the package version listed in the HPE Ezmeral Runtime Enterprise UI.

To prevent yum from updating Kubernetes packages as part of operations such as rebooting a host, on each host, ensure that the yum update command is prevented from updating the Kubernetes repo:

  1. Open the following file in an editor:

    /etc/yum.repos.d/bd-kubernetes.repo
  2. The parameter enabled=1 indicates that updates are enabled. To disable updates, change the parameter to enabled=0.

If you want to use yum to update other packages, run the yum update command without the -y option so that you can individually deny any Kubernetes packages updates that show as available.

Performing System Maintenance

  1. If this host is a Kubernetes host, see Preventing Unintended Updates of Kubernetes Packages
  2. Enter Lockdown mode, as described in Lockdown Mode.
  3. Upgrade the Controller host, then reboot the host, and then wait for all services to come back up in the Services tab of the Platform Administrator Dashboard (see Dashboard - Platform Administrator).
  4. If you have platform High Availability enabled, then repeat Step 3 for the Shadow Controller host.
  5. If you have platform High Availability enabled, then repeat Step 3 for the Arbiter host.
  6. Upgrade one Worker host, then reboot that Worker, and then wait for all services to come back up in the Services tab of the Platform Administrator Dashboard.

    If this host is a GPU host, NVIDIA GPU drivers must be reinstalled after OS Kernel updates because the NVIDIA kernel module has a kernel interface layer that must be compiled specifically for each kernel. To reinstall the NVDIA GPU drivers, see Steps 8 to 11 in GPU Driver Installation.

  7. Repeat Step 6 for each remaining Worker host, ensuring that you perform the entire process on one host at a time.
  8. After all hosts have been fully rebooted, exit Lockdown mode and then perform the tests described in Validating the Installation to verify that the platform is functioning normally.

If you want to remove RPM packages after HPE Ezmeral Runtime Enterprise is installed, then be sure not to remove any required packages.

The Kubernetes RPMs file for air gap installations contains the required RPMs. See Configuring Air Gap Kubernetes Host Settings. If you need a separate list of RPMs for this version of HPE Ezmeral Runtime Enterprise, contact your Hewlett Packard Enterprise Support representative.