Upgrading Kubernetes

Before Upgrading Kubernetes

There are some situations in which you will need help from Hewlett Packard Enterprise support or in which you must perform workaround tasks.

HPE Ezmeral Data Fabric on Kubernetes is Deployed
If you are not upgrading Data Fabric, do the following:
  1. Bring the Data Fabric cluster to offline status by performing the steps in Shutting Down a Data Fabric Cluster to make the cluster offline.
  2. Upgrade the Kubernetes version by performing steps of Upgrade Procedure in this page.
  3. After you perform the Upgrade, Restart the Data Fabric cluster by performing the steps in Restarting the Data Fabric Cluster.
Kubeflow Add-On is Deployed

If you are upgrading a Kubernetes cluster in HPE Ezmeral Runtime Enterprise, the Kubeflow add-on must be version 1.6 or greater.

To check your version of Kubeflow, look in the Kubeflow dashboard. See Accessing the Kubeflow Dashboard.

If your Kubernetes version is lower than 1.6, upgrade the Kubeflow and Istio add-ons before proceeding with the Kubernetes cluster upgrade. See Upgrading Kubernetes Add-Ons.

Istio Add-On is Deployed

When you deploy Istio add-on on Kubernetes cluster, one or more Worker nodes will fail to upgrade the Kubernetes version. The Kubernetes version upgrade fails with the following errors:

  • Warning: one or more workers failed to upgrade on the Kubernetes Cluster screen.
  • Upgrade error: Failed to drain node error at the individual Kubernetes Host Status screen

To resolve the errors, see EZCP-1608 in Issues and Workarounds

Pods use PVCs provisioned by the CSI driver

If pods on this cluster use a persistent volume claim (PVC) provisioned through HPE CSI driver 1.0.x or 1.1.x, before you upgrade from Kubernetes 1.18.x, upgrade the HPE CSI driver to version 1.2.5-1.0.5. For instructions, see Upgrading the CSI Plug-In.

If you do not upgrade the CSI driver, pods fail to come up, and the Kubernetes Cluster screen displays the message: one or more workers failed to upgrade.

Upgrade Procedure

In the Kubernetes Clusters screen (see The Kubernetes Clusters Screen), clicking the Upgrade Kubernetes icon (up arrow) in the Actions column opens the Kubernetes Upgrade popup.



To upgrade the Kubernetes version:

  • Use the Upgrade Version pull-down menu to select the new version of Kubernetes to install. This menu is disabled if no new version of Kubernetes is available.
  • Use the Upgrade Percentage pull-down menu to specify the number of Kubernetes Worker nodes that will be upgraded at any one time. The default selection is 20%. For example, if the cluster has five Worker nodes, then they will be upgraded one at a time; a cluster with 15 Worker nodes will be upgraded 3 at a time, and so forth. This menu is disabled if no new version of Kubernetes is available.
    NOTE
    For Kubernetes Data Fabric clusters in HPE Ezmeral Runtime Enterprise 5.3.5 or later, the percentage setting is ignored. Clusters are upgraded one node at a time.

The status of Worker hosts changes to Upgrading during the upgrade process. During the upgrade:

  • The Master hosts are upgraded first.
  • If any of the Master hosts fails to upgrade, then the upgrade will be rolled back, the Kubernetes version will remains unchanged, and the status of the Kubernetes cluster will change to WARNING.
  • The Worker hosts are upgraded in batches according to the percentage setting after the Master hosts have been successfully upgraded.
  • If any Worker host fails to upgrade, then its status will change to UPGRADE ERROR and its Kubernetes version remains unchanged.
  • You can retry the upgrade on any failed hosts by clicking the Retry Upgrade button.

When upgrading Kubernetes or HPE Ezmeral Runtime Enterprise:

  • Kubernetes clusters: This is a "rolling" process where containers are upgraded sequentially.
  • Control plane (Controller, Shadow, and Arbiter hosts): This is not a rolling process, and upgrades do not affect running Kubernetes clusters or workloads. However, the authentication proxy will be down for a short period during the upgrade process, which will interrupt access to Kubernetes clusters. Access will be restored shortly.

See General Kubernetes Application/Deployment Issues for assistance if you experience any errors while upgrading Kubernetes.