Upgrading Kubernetes Add-Ons

Use this procedure to upgrade the Kubernetes add-ons and to install new required add-ons on existing Kubernetes clusters in HPE Ezmeral Runtime Enterprise.

Prerequisites

Required access rights: Platform Administrator
You have Upgraded the platform software.
Exit Lockdown mode. See Exiting Lockdown Mode.
You have upgraded HPE Ezmeral Data Fabric on Kubernetes, if applicable.

About this task

In this procedure, you execute a script on each Kubernetes cluster. The script lists the deployed add-ons compared to the Kubernetes manifest, upgrades deployed add-ons, and installs new required add-ons.

IMPORTANT

If you are upgrading from HPE Ezmeral Runtime Enterprise 5.3.1, the Kubernetes add-on upgrade steps for GPU hosts are different. See the issues and workarounds in the Release Notes.

If your environment includes HPE Ezmeral Data Fabric on Kubernetes clusters, ensure that you have ugraded HPE Ezmeral Data Fabric on Kubernetes before you upgrade the Kubernetes add-ons in this procedure.

Procedure

On the primary HPE Ezmeral Runtime Enterprise controller, change to the directory: /opt/hpe/kubernetes/tools
List all deployed add-ons compared to the add-ons in the manifest by executing the script using the -t (test) and --required-only parameters.
In the following example, the HTTPS is enabled on the controller at IP address 192.0.2.5. The -x parameter specifies that HTTPS is used. The command specifies cluster name my-k8s-cluster. The -f parameter specifies the path to a JSON file that contains the username and password to use to connect to the controller host, and directs the script to use the Site Administartor tenant when updating the Kubernetes manifest file. For more information about the credentials file, see Kubernetes Add-On Upgrade Script.
```
python k8s_addon_upgrade.py -c 192.0.2.5 -f /tmp/cred.json -x -k my-k8s-cluster --required-only -t
```
The output of the command lists all of the manifest add-ons (both required and optional), and their versions, and all the add-on versions that are currently deployed on the cluster.

By executing the script with the -t (or --dry-run) parameter, you can compare the deployed and manifest versions of optional add-ons, such as Istio, to determine whether or not you want to upgrade that add-on. If you have an applications that requires an earlier versions of an add-on, for example, you might not want to include that add-on when you execute the script to perform the upgrade.
In the following example output, the argocd add-on is new to the my-k8s-cluster cluster, so there is no value displayed for the deployed version.
```
Cluster my-k8s-cluster required add-ons info:
  argocd:
    deployed version: , tools: 
    manifest version: 1.8.4-1, tools: 0.4
...
  hpecp-agent:
    deployed version: 1.1.2-1, tools: 0.1
    manifest version: 1.1.5-4, tools: 0.4
...
dry run: upgrade add-ons ...<list-of-add-ons>... on cluster my-k8s-cluster
Done
```
When you execute the script without the -t (or --dry-run) parameter, the script installs the manifest versions of the add-ons you specify in the command. The --required-only parameter enables you to specify only the required add-ons without having to list each required add-on in the command individually.
(Upgrading from HPE Ezmeral Runtime Enterprise versions prior to 5.4.0 only.)
(Optional) If you have enabled the Spark Operator, you will see the Spark Operator add-on:
```
picasso-compute:
deployed version: picasso-1.4.1-drop7-43-2, tools: 0.4
manifest version: picasso-1.5.0-P150RC5-65-0, tools: 0.5
```
Delete the Spark Operator add-on using the following command:
```
python k8s_addon_upgrade.py -c 192.0.2.5 -f /tmp/cred.json -x -k my-k8s-cluster --cleanup-k8scluster-record
```
Peform the upgrade by executing the script with the -t parameter omitted.
For example:
```
python k8s_addon_upgrade.py -c 192.0.2.5 -f /tmp/cred.json -x -k my-k8s-cluster --required-only
```
The add-on upgrade process can take more than 30 minutes to complete.
Verify that the add-ons are updated.

Execute the script again with the -t and --required-only parameters. The output should show that the deployed versions and manifest versions are the same.

If any add-on upgrades or installations have failed, you can run the script with the --refresh option to retry the operation. For information, see Kubernetes Add-On Upgrade Script.
(Optional) Upgrade optional add-ons, such as Istio.

After you upgrade the required add-ons on each Kubernetes cluster, you can use the same script to upgrade optional add-ons on each cluster.
The following example shows using the -a parameter to update only the Istio add-on.
```
python k8s_addon_upgrade.py -c 192.0.2.5 -f /tmp/cred.json -x -k my-k8s-cluster -a istio
```
(Optional) If you have HPE Ezmeral Data Fabric on Kubernetes tenant add-on, in the Data Fabric cluster, you will see the information shown in the following example:
```
picasso-tenant:
  deployed version: picasso-1.4.1-drop7-43-2, tools: 0.4
  manifest version: picasso-1.5.0-P150RC8-68-0, tools: 0.5
```
Proceed to Upgrading Kubernetes to a Later Version.