Upgrading to HPE Ezmeral Runtime Enterprise 5.6.x

Upgrade Paths

You can upgrade directly from the following previous versions of HPE Ezmeral Runtime Enterprise only:

• 5.6.2
• 5.6.3

Upgrading Kubernetes Bundles

Beginning with HPE Ezmeral Runtime Enterprise 5.5.0, HPE decouples the upgrade of the HPE Ezmeral Runtime Enterprise platform from Kubernetes-related components.

Air-Gapped Kubernetes Deployment

An air-gapped Kubernetes deployment refers to a deployment in which Kubernetes hosts, clusters, and tenants do not have connectivity to the Internet. Air-gapped deployments are also called disconnected sites or dark sites.

For information about requirements for air-gapped environments, see Kubernetes Air-Gap Requirements.

Impact on Workloads During Platform Upgrade

During an upgrade of the HPE Ezmeral Runtime Enterprise, workloads are affected as follows:

Kubernetes Clusters and workloads

• New Kubernetes clusters cannot be created during the control plane upgrade process. Workloads running on existing Kubernetes clusters or within a tenant namespace will be impacted when the Container Runtime is restarted. Service endpoints will be inaccessible due to the unavailability of the authentication proxy.
• Any new service points ( e.g. NodePort services) that are in flight for port remapping via a Gateway host will be queued up until the control plane upgrade completes. Affected service endpoints will be remapped via a Gateway host and made available for users when the control plane resumes operating.
• All existing service endpoints across one or more Kubernetes clusters and/or tenants will continue to serve traffic during the control plane upgrade except for a brief (approximately 5 seconds) interval while the haproxy component restarts during the Gateway upgrade.
• Kubernetes users across all roles (e.g. Cluster Administrator or Member) will not be able to interact with the Kubernetes API server via either kubectl, the web interface, or any other means during the upgrade process. Terminal access to Kubernetes cluster is still possible via SSH, assuming that administrators can access Kubernetes nodes.
• Cluster manipulation activities such as upgrading existing kubernetes clusters or expanding master/worker nodes are not allowed until the upgrade completes.

Non-Kubernetes virtual nodes and workloads

• Existing non-Kubernetes virtual nodes/containers will be down and/or inaccessible during the upgrade process because all Docker processes will be “paused” (or stopped) for all “active” virtual clusters before the upgrade starts.
• All orchestration activities (e.g. expand/shrink, ActionScripts, Gateway port-mapping, node migration, etc.) will be unavailable until the upgrade completes.
• Tasks are not queued during the upgrade; you must reinitiate any such tasks after the upgrade.

Upgrade Process Summary

The following is a summary of the process to upgrade to HPE Ezmeral Runtime Enterprise and its related components. Detailed procedures are provided in the tasks linked to by this summary.

1. Before you upgrade the platform, you might need to upgrade host software to a version that is supported both on the HPE Ezmeral Runtime Enterprise version you are running and on the HPE Ezmeral Runtime Enterprise version to which you are upgrading.
   1. Upgrade host operating system software on each host.
   2. If using an air-gapped environment, Configure or change Kubernetes air gap settings.
   3. Upgrade Kubernetes versions on each Kubernetes cluster. Because Kubernetes versions must be upgraded one at a time, you might need to perform multiple upgrades.

   Other tasks you perform before upgrading the software include obtaining the software and executing pre-check scripts on all hosts.

2. If your deployment includes hosts that have GPUs, you must remove the hosts from the Kubernetes clusters and remove the hosts from HPE Ezmeral Runtime Enterprise. You complete this task in Before Upgrading the Platform.
   IMPORTANT

   HPE Ezmeral Runtime Enterprise adds support for MIG-enabled GPUs. For all GPUs to be recognized by the system after the upgrade, all hosts that have GPUs must be removed from HPE Ezmeral Runtime Enterprise before the upgrade, and then added back to the configuration after the upgrade process is complete. This requirement applies to all GPUs, including those GPUs are that are not MIG-enabled.

3. If your environment includes HPE Ezmeral Data Fabric on Kubernetes clusters, proceed to Upgrade HPE Ezmeral Data Fabric on Kubernetes.

   If your environment implements Embedded Data Fabric, there are no specific upgrade tasks to perform for that feature. You can procede to upgrading the platform software.

   IMPORTANT

   A deployment of HPE Ezmeral Runtime Enterprise can include one Data Fabric cluster that implements HPE Ezmeral Data Fabric on Kubernetes, or the deployment can include an Embedded Data Fabric, but not both (see Storage). If this deployment already includes an Embedded Data Fabric, do not attempt to add another Kubernetes Data Fabric cluster.

4. Upgrade the platform software on the Controller host.
5. If you want to install Falco Kernel modules on your Kubernetes hosts, consider installing them before you upgrade the Kubernetes add-ons. See Install the Falco Kernel modules on each Kubernetes host.
6. If you want to upgrade Kubernetes add-ons, see Upgrade Kubernetes add-ons on each Kubernetes cluster.
7. If you want to upgrade Kubernetes clusters to later versions, see Upgrade Kubernetes clusters to later versions of Kubernetes. HPE Ezmeral Runtime Enterprise supports multiple versions of Kubernetes. Depending on the applications you have installed or plan to install, you might want to upgrade Kubernetes to a later version than the version you installed before the platform upgrade.
8. If you are using the RHEL 8, find the Data Fabric hosts and reboot those hosts.
   NOTE You can see the Data Fabric hosts by looking at the Kubernetes Hosts page that lists all the hosts. The Data Fabric hosts have Data fabric: true tag.
9. Restart HPE Ezmeral Data Fabric on Kubernetes cluster. See Restarting the Data Fabric Cluster for details.
10. On the Controller node, re-establish the tenant mount, using the following commands:

    ERTS_PATH=/opt/bluedata/common-install/bd_mgmt/erts-*/bin
    NODETOOL=/opt/bluedata/common-install/bd_mgmt/bin/nodetool
    NAME_ARG=`egrep '^-s?name' $ERTS_PATH/../../releases/1/vm.args`
    RPCCMD="$ERTS_PATH/escript $NODETOOL $NAME_ARG rpcterms"
    $RPCCMD bd_hypervisor_controller_common redo_tenant_storage_mounts

11. Verify the upgrade.
12. If you want to add hosts that include GPUs, or you want to upgrade Kubernetes to later versions, do those tasks as part of the Post Upgrade Tasks.