Upgrading HPE Ezmeral Unified Analytics Software

Provides the steps to upgrade HPE Ezmeral Unified Analytics Software.

Unified Analytics administrators can upgrade HPE Ezmeral Unified Analytics Software to the latest version of the product.
REMEMBER
  • To upgrade HPE Ezmeral Unified Analytics Software to version 1.5.0, contact HPE Support.
  • To upgrade HPE Ezmeral Unified Analytics Software from version 1.5.0 to 1.5.2, follow the instructions outlined in this topic.

The upgrade process typically lasts up to 90 minutes. During the upgrade, the infrastructure services are upgraded first, followed by applications upgrade. You can monitor the upgrade as it progresses. While the upgrade is in progress, you cannot access the HPE Ezmeral Unified Analytics Software UI; however, the UI automatically loads and becomes accessible when the upgrade completes.

Complete the following steps to upgrade HPE Ezmeral Unified Analytics Software:

Complete Prerequisites

Before upgrading HPE Ezmeral Unified Analytics Software to the new version, ensure the following prerequisites are met:
  • You have administrative access to the cluster running the current version of HPE Ezmeral Unified Analytics Software.
  • Download the HPE Ezmeral Coordinator upgrade package (ezua-upgrade-package-to-<version-number>-commit-chksum.tgz) from MY HPE SOFTWARE CENTER customer portal.
  • To upgrade, you need the kubeconfig files for the HPE Ezmeral Coordinator and workload clusters. You can get the kubeconfig files through the Download Kubeconfig button on the last UI installation screen from the previous installation.


    Alternatively, you can SSH into the HPE Ezmeral Coordinator and workload nodes to get the kubeconfig files.
    NOTE
    If you need to identify the HPE Ezmeral Coordinator nodes, run the following command: 
    kubectl get nodes --kubeconfig <ezmeral-coordinator-kubconfig-file>
  • Verify that pods are not in the Pending state. To check if the pods are in the Pending state, run the following command on the workload cluster:
    kubectl get pod -A | grep Pending
  • Verify that the workload cluster has CPU and memory resources available, with approximately 20 free CPU units in the cluster. You can check the CPU and memory resources by running the following commands on the workload cluster:
    Actions Commands
    Check the CPU and memory resources on a particular worker node. 
    kubectl describe node ${NodeName}
    Check the CPU resources on all worker nodes. 
    kubectl describe node -l node-role.kubernetes.io/worker | grep cpu | grep -v cpu-
    Check the CPU resources on a particular worker node. 
    kubectl describe node ${NodeName} | grep cpu | grep -v cpu-
    Check the memory resources on all worker nodes. 
    kubectl describe node -l node-role.kubernetes.io/worker | grep memory | grep -v MemoryPressure
    Check the memory resources on a particular worker node. 
    kubectl describe node ${NodeName} | grep memory | grep -v MemoryPressure
  • Verify that the existing ezaddon resources in the Unified Analytics cluster are in the Installed state. To check the status of all the ezaddon resources, run the following command on the workload cluster:
    kubectl get ezad -A
NOTE

If the verification for prerequisites fails, contact HPE support.

Upgrade HPE Ezmeral Coordinator Cluster

  1. Use scp to manually copy the downloaded upgrade package (ezua-upgrade-package-to-<version-number>-commit-chksum.tgz) to the HPE Ezmeral Coordinator cluster control plane node.
  2. Extract the upgrade package. For example:
    [ouxi@m2-lr1-dev-vm210015 ~]$ tar xvf ezua-upgrade-package-to-<version-number>-commit-chksum.tgz
upgrade/
upgrade/IMAGES
upgrade/ezfab-release.tgz
upgrade/ezfabricctl_darwin_amd64
upgrade/ezfabricctl_linux_amd64
upgrade/ezua-system-<version>.tgz
upgrade/ezfab-upgrade.sh
upgrade/VERSION
[ouxi@m2-lr1-dev-vm210015 ~]$ ls -al
total 278660
drwx------. 4 ouxi ldap       182 Sep 26 22:40 .
drwxr-xr-x. 4 root root        30 Sep 26 16:19 ..
-rw-------. 1 ouxi ldap        18 Sep 26 16:10 .bash_logout
-rw-------. 1 ouxi ldap       141 Sep 26 16:10 .bash_profile
-rw-------. 1 ouxi ldap       376 Sep 26 16:10 .bashrc
-rw-------. 1 ouxi ldap 285124183 Sep 26 22:40 ezua-upgrade-package-to-<version-number>-commit-chksum.tgz
drwx------. 3 ouxi ldap        33 Sep 26 16:21 .kube
-rw-------. 1 ouxi ldap    207894 Sep 26 16:21 mgmt-bootstrap.log.xtrace
drwx------. 2 ouxi ldap       176 Sep 25 18:34 upgrade
  3. The upgrade directory includes the ezfab-upgrade.sh script, which is used to upgrade HPE Ezmeral Coordinator. To run the ezfab-upgrade.sh script, prepare the kubeconfig file at $HOME/.kube/ezfab.config. For example:
    [ouxi@m2-lr1-dev-vm210015 ~]$ cd .kube/
[ouxi@m2-lr1-dev-vm210015 .kube]$ ls
cache  config
[ouxi@m2-lr1-dev-vm210015 .kube]$
[ouxi@m2-lr1-dev-vm210015 .kube]$
[ouxi@m2-lr1-dev-vm210015 .kube]$ cp config ezfab.config
[ouxi@m2-lr1-dev-vm210015 .kube]$
[ouxi@m2-lr1-dev-vm210015 .kube]$
[ouxi@m2-lr1-dev-vm210015 .kube]$ ls -al
total 16
drwx------. 3 ouxi ldap   53 Sep 26 22:40 .
drwx------. 4 ouxi ldap  182 Sep 26 22:40 ..
drwx------. 4 ouxi ldap   35 Sep 26 16:21 cache
-rw-------. 1 ouxi ldap 5671 Sep 26 16:21 config
-rw-------. 1 ouxi ldap 5671 Sep 26 22:40 ezfab.config
  4. Navigate to the upgrade directory and manually run the ezfab-upgrade.sh script to trigger the HPE Ezmeral Coordinator upgrade. For example:
    [ouxi@m2-lr1-dev-vm210015 .kube]$ cd
[ouxi@m2-lr1-dev-vm210015 ~]$ ls
ezua-upgrade-package-to-<version-number>-commit-chksum.tgz  mgmt-bootstrap.log.xtrace  upgrade
[ouxi@m2-lr1-dev-vm210015 ~]$ cd upgrade/
[ouxi@m2-lr1-dev-vm210015 upgrade]$ ls
ezfab-release.tgz  ezfabricctl_darwin_amd64  ezfabricctl_linux_amd64  ezfab-upgrade.sh  ezua-system-<version>.tgz  IMAGES  VERSION
[ouxi@m2-lr1-dev-vm210015 upgrade]$
[ouxi@m2-lr1-dev-vm210015 upgrade]$ ./ezfab-upgrade.sh
Check OS ...
 
Running ezfabricctl orchestrator upgrade...
 
No resources found
INFO Upgrading Ezmeral coordinator cluster/components
INFO ezfabricctl version: 1.3.0-1a6b8d0
INFO Build date: 2024-09-25T23:56:04Z
INFO Release Package: ezfab-release.tgz
INFO Deploy Target:
INFO Deploy Env:
INFO Running precheck
INFO Upgrading Ezmeral coordinator cluster
...
INFO upgrade addons: [longhorn]
INFO Started monitoring the status of service addons (EzkfUpgrade)
INFO Service Addons (EzkfUpgrade) are not upgraded yet. Waiting...
INFO Service Addons (EzkfUpgrade) are not upgraded yet. Waiting...
INFO Service Addons (EzkfUpgrade) are not upgraded yet. Waiting...
INFO Service Addons (EzkfUpgrade) are not upgraded yet. Waiting...
INFO Service Addons (EzkfUpgrade) are not upgraded yet. Waiting...
INFO Upgraded service addons in Ezmeral coordinator cluster
INFO Installing EzUA plugin CR: ezua-plugin-<version>-75f3ef1.yaml
INFO Successfully upgraded Ezmeral coordinator cluster/components
    The output of the ezfab-upgrade.sh script displays the upgrade progress. A successful upgrade is indicated by the message: "INFO Successfully upgraded Ezmeral coordinator cluster/components". Some pods may not enter the Running state immediately after the script finishes.
  5. Wait and monitor all pods to verify they are all in the Running state. For example:
    [ouxi@m2-lr1-dev-vm210015 upgrade]$ kubectl get pod -A
NAMESPACE                 NAME                                                     READY   STATUS              RESTARTS       AGE
capi-system               capi-controller-manager-69b84d5d98-knns6                 1/1     Running             5 (22m ago)    6h59m
cert-manager              cert-manager-5b99f586f8-rnw6t                            1/1     Running             0              37m
cert-manager              cert-manager-cainjector-b8cdfd94f-cqjnw                  1/1     Running             0              37m
cert-manager              cert-manager-webhook-58c8649c5c-vnm2n                    1/1     Running             0              37m
ezaddon-system            ezaddon-controller-manager-7865c6f86b-ktcxx              1/1     Running             1 (22m ago)    39m
ezkf-mgmt                 op-v2-upgrade-addonproc-ezkf-mgmt-f42q7                  1/1     Running             0              3m13s
ezkf-system               ezfabric-controller-manager-6b95849759-nq67n             2/2     Running             3 (22m ago)    35m
ezkf-system               ezkf-bootstrap-controller-manager-687875bc97-tlt7k       2/2     Running             3 (22m ago)    6h59m
ezkf-system               ezkf-control-plane-controller-manager-7bd7f9f876-85bl7   2/2     Running             4 (22m ago)    6h58m
ezkf-system               ezkf-infrastructure-manager-57b6675545-82kc8             2/2     Running             4 (24m ago)    6h58m
ezua-demo                 ezua-demo-controller-manager-76d8dc858f-nbmhh            2/2     Running             2 (5h4m ago)   6h28m
ezua-demo                 op-clustercreate-ezua-demo                               1/1     Running             0              6h29m
ezua-demo                 op-plugin-upgrade-ezua-demo                              0/1     ContainerCreating   0              27s
ezua-demo                 w-op-workload-deploy-ezua-demo                           1/1     Running             0              6h28m
kube-system               calico-kube-controllers-5696576df5-s99dl                 1/1     Running             0              7h3m
kube-system               calico-master-dgq5n                                      1/1     Running             0              7h3m
kube-system               calico-typha-66c46875c4-5xktv                            1/1     Running             0              7h3m
kube-system               coredns-d4544c989-5vlbq                                  1/1     Running             0              7h2m
kube-system               coredns-d4544c989-hr2zj                                  1/1     Running             0              7h2m
longhorn-system           csi-attacher-67578d8dc7-q9pqd                            1/1     Running             0              19m
longhorn-system           csi-provisioner-59fd5dfcff-bwvgl                         1/1     Running             0              18m
longhorn-system           csi-resizer-8686444cc9-66c4v                             1/1     Running             0              17m
longhorn-system           csi-snapshotter-c49d97966-99txb                          1/1     Running             1 (15m ago)    16m
longhorn-system           engine-image-ei-6435403c-sl8x6                           1/1     Running             0              27m
longhorn-system           instance-manager-0ee8f243f69bf3a6841efc3b74cdcf05        1/1     Running             0              23m
longhorn-system           longhorn-csi-plugin-lsm7v                                3/3     Running             0              15m
longhorn-system           longhorn-driver-deployer-56b8686647-v9nts                1/1     Running             0              28m
longhorn-system           longhorn-manager-95r5t                                   1/1     Running             0              27m
longhorn-system           longhorn-ui-7dbff4bb69-wwbkn                             1/1     Running             0              28m
mgmt-cluster-ops-system   mgmt-cluster-ops-controller-manager-7596655d45-kfqml     2/2     Running             3 (22m ago)    33m
[ouxi@m2-lr1-dev-vm210015 upgrade]$
  6. Verify that all ezaddons are in the installed state. For example:
    
[ouxi@m2-lr1-dev-vm210015 ~]$ kubectl get ezad -A
NAMESPACE        NAME                      CLUSTER      INSTALL   STATE       PHASE   REASON   PROGRESS
ezaddon-system   cert-manager              in-cluster   true      installed                   
ezaddon-system   ezkf                      in-cluster   true      installed                   
ezaddon-system   in-cluster-csi-detector   in-cluster   true      installed                   
ezaddon-system   longhorn                  in-cluster   true      installed                   
[ouxi@m2-lr1-dev-vm210015 ~]$

Verify HPE Ezmeral Coordinator Cluster Upgrade Success

  • Review the logs stored in the ./log directory to ensure there are no errors or warnings.
  • Verify that there are no errors or warnings in the console output during the upgrade process.
  • Verify that the ezkf-agent status is in the active (running) state. To verify, run:
    systemctl status ezkf-agent
  • Verify that the Kubernetes node status is in the Ready state. To verify, run:
    kubectl get nodes

If the upgrade fails, contact HPE Support.

Download Workload Cluster Upgrade Bundle

You can either automatically or manually download the upgrade bundle required to upgrade Unified Analytics workload cluster. In a connected environment, you can enable an automatic download or perform a manual download. In an air-gapped environment, you must manually download the bundle.

(Air-gapped environment only) Before you complete the following steps, use the air-gap utility to download all the images required for the Unified Analytics version you want to upgrade to and push those images into the air-gap registry. To learn about how to use the air-gap utility, see Using the Air Gap Utility.

To download the upgrade bundle for both the air-gapped and connected environment, complete the following steps:
  1. In the left navigation bar, click Administration → Settings.
  2. Click Updates → Download Updates.
  3. Enter Image Name.

    To find the the upgrade bundle images, see Upgrade Bundle Images.

    An example of the image name for:
    • An air-gapped environment: <RegistryURL>/ezua/ezua-system-bundle:<image-tag>
    • The connected environment: marketplace.us1.greenlake-hpe.com/ezua/ezua/ezua-system-bundle:<image-tag>.
  4. Click Download.
  5. View the downloaded image in the table.
    NOTE
    To remove an image, click Delete.

  6. After successfully downloading a new image, wait for a few minutes (at most five minutes) to see the upgrade bundle on the Available Updates table as EZUA System.

  7. Once you see the EZUA System upgrade bundle, continue to the Trigger Workload Cluster Upgrade from UI section.

In a connected environment, you can upgrade Unified Analytics by enabling the automatic downloads of the upgrade bundle. After the upgrade bundle is downloaded, you can view it on the Available Updates table.

To automatically download the upgrade bundle, complete the following steps:
  1. In the left navigation bar, click Administration → Settings.
  2. Click Updates → Download Updates.
  3. To enable the automatic download of the upgrade bundle, toggle Enable.
    NOTE
    To disable the automatic download of the upgrade bundle, toggle Disable. When you disable the automatic download of the upgrade bundle in the connected environment, you must manually downoad the upgrade bundle to upgrade to the latest version of the product.
  4. After enabling the automatic downloads, when an upgrade is available for a new version, HPE Ezmeral Unified Analytics Software displays the upgrade bundle on the Available Updates table as EZUA System.
  5. Once you see the EZUA System upgrade bundle, continue to the Trigger Workload Cluster Upgrade from UI section.

Trigger Workload Cluster Upgrade from UI

Once you see the EZUA System upgrade bundle, you can click the bundle to view details. In the Details dialog box, you can see the name, description, the current version of the Unified Analytics, and the new available version for Unified Analytics. Once you see the new available versions for the upgrade, you can perform the following actions by clicking on the Actions menu.
To upgrade Unified Analytics immediately, complete the following steps:
  1. Click Update Now in the Actions menu. This will open an Update Now dialog box and you can compare the current and new available versions of Unified Analytics for upgrade.
  2. Click Update Now to immediately start the Unified Analytics upgrade. Wait for the upgrade to be in the In Progress status.
    NOTE
    You cannot cancel the upgrade once it is in the In Progress status.
  3. To monitor the upgrade process, see Monitor Workload Cluster Upgrade Progress.
To schedule the upgrade for later, complete the following steps:
  1. Click Schedule Update in the Actions menu. This will open a Schedule Update dialog box and you can compare the current and new available versions of Unified Analytics for upgrade.
  2. Select a date and time to schedule the update. Wait for the upgrade to be in the Scheduled status.
  3. Once an upgrade is in the Scheduled status, you can perform the following actions from the Actions menu.
    Cancel

    You can cancel the scheduled upgrade any time before upgrade starts or if upgrade is not in the In Progress status yet.

    Reschedule

    You can reschedule the scheduled upgrade any time before upgrade starts or if upgrade is not in the In Progress status yet.

    Update

    You can upgrade Unified Analytics immediately even though it has been scheduled for a later date and time.

  4. To monitor the upgrade process, see Monitor Workload Cluster Upgrade Progress.

Monitor Workload Cluster Upgrade Progress

You can use the workload cluster to monitor the upgrade as it progresses. You can also use the HPE Ezmeral Coordinator cluster to debug errors and collect log files.

To monitor the upgrade progress, run the following commands on the Unified Analytics workload cluster:
Actions Commands Descriptions
View the upgrade CR. 
kubectl get EzkfWorkloadUpgrade -A -o yaml
  • Retrieves the EzkfWorkloadUpgrade custom resource (CR) in YAML format.
  • Allows you to view detailed information about the EzkfWorkloadUpgrade CR, including its configuration and current state.
  • Enables you to troubleshoot and debug a failure reason.
List the ezaddon resources. 
kubectl get eza -A
  • Retrieves a list of all the ezaddon resources from all namespaces.
  • Allows you to view the installed version of each ezaddon resource.
Monitor the current status of the ezaddon resources. 
kubectl get ezad -A
  • Retrieves a list of all the ezaddon resources from all namespaces.
  • Allows you to view the current status of each ezaddon resource, such as installed, upgrading, failed, and others.
Monitor the status of the ezaddon resources in real-time. 
kubectl get ezad -A -w
  • Retrieves a list of all the ezaddon resources from all namespaces and continuously watches for any changes or updates to the ezaddon resources.
  • Allows you to monitor the status and changes of the ezaddon resources in real-time.
Monitor the bootstrap pod. 
kubectl get pod -n ezaddon-system
  • Monitor the bootstrap pod in the ezaddon-system namespace.
  • Allows you to track the progress of the upgrade and identify any potential issues or failures early on.
Monitor the total time for the upgrade process. 
kubectl get EzkfWorkloadUpgrade -A -o yaml | grep Time
  • Extracts the time required to complete the upgrade process from the YAML output.
To debug and collect log files, run the following commands on the HPE Ezmeral Coordinator cluster:
  1. To get a list of pods in the HPE Ezmeral Coordinator cluster, run:
    kubectl get pod -A
  2. In the list of pods, locate the upgrade pod named upgrade-ezua-upgrader-<a-b-c-wxyz-namespace>.
  3. To get log files, run:
    kubectl logs -f upgrade-ezua-upgrader-<a-b-c-wxyz-namespace> -n <namespace>

Verify Workload Cluster Upgrade Success

If the upgrade is successful, the HPE Ezmeral Unified Analytics Software UI loads automatically. If it does not load automatically, manually reload and sign in.

To verify the sucessful upgrade:
  • Click the <username> dropdown at the top right. You will see that the Build Version for the Unified Analytics cluster has been updated to the new version.

  • View the upgrade details in the Update History table. You can click EZUA System to view the upgrade details. In the Details dialog box, you can see the name, description, and versions of Unified Analytics.

NOTE

If you are unable to access the HPE Ezmeral Unified Analytics Software UI after 90 minutes, monitor the upgrade progress and check the logs from the HPE Ezmeral Coordinator cluster. For details, see Monitor Workload Cluster Upgrade Progress.

If the upgrade fails, contact HPE Support.