Upgrading HPE Ezmeral Data Fabric on Kubernetes

This procedure describes upgrading HPE Ezmeral Data Fabric on Kubernetes clusters as part of upgrading HPE Ezmeral Runtime Enterprise. This task does not apply to Embedded Data Fabric implementations.

Prerequisites

NOTE

This task is applicable only when upgrading to HPE Ezmeral Runtime Enterprise 5.4.1 or later only.

If you are upgrading to HPE Ezmeral Runtime Enterprise 5.4.0, contact your Hewlett Packard Enterprise support for upgrade assistance.

Required access rights: Platform Administrator

About this task

If your environment implements Embedded Data Fabric, there are no specific upgrade steps to complete. Skip this task and proceed to Upgrading Kubernetes Add-Ons.

Procedure

On the Kubernetes master node, perform the following steps:

IMPORTANT If you are using Kubernetes 1.19.x or later, skip this step and proceed to the next step.
If you are using Kubernetes 1.18.x, execute these commands to download and install kubectl component for Kubernetes 1.19.x. You must only download the kubectl, and make sure you do not upgrade to Kubernetes 1.19.x.
```
cd /tmp
curl -LO https://dl.k8s.io/release/v1.19.0/bin/linux/amd64/kubectl
mv /usr/bin/kubectl /usr/bin/kubectl.orig
chmod 777 kubectl
mv kubectl /usr/bin/kubectl
```
If you are using Kubernetes 1.17.x or older, contact Hewlett Packard Enterprise support.
Upgrade HPE Ezmeral Data Fabric on Kubernetes as follows:
1. If you are using Centos 7, install python36 using following command:
```
yum install -y python36
```
  NOTE You must remove the python36 at the end of this procedure.
2. Make sure you get HPE Ezmeral Data Fabric on Kubernetes 1.5.1, using following commands:
```
git clone https://github.com/HPEEzmeral/df-on-k8s.git
cd df-on-k8s/bootstrap/p1.5.1
sed -i 's/BOOTSTRAP_PYTHON=.*$/BOOTSTRAP_PYTHON="python3"/' bootstrap.sh
```
3. In the HPE Ezmeral Data Fabric on Kubernetes repository that you cloned in step 2 b., run the following command:
```
./bootstrap.sh upgrade --std_csimount
```
  You can see the following prompts:
```
WARNING: Updating the CSI driver is a disruptive operation. All pods using CSI will need to be restarted manually. If the objectstore pod is running, it will also need to be restarted manually.yes
>>> Update Ezmeral Data Fabric CSI driver? (yes/no) [yes]: yes
/bin/sh: no: No such file or directory 
/bin/sh: no: No such file or directory 
```
  The preceding step is optional. If you update the Data Fabric CSI driver, then anything that uses CSI, such as notebooks, must be restarted. However, it is good to upgrade CSI drivers, as you are upgrading the entire system, and most of the pods will be restarted anyway.
```
Use Airgapped Docker Registry? Note: All bootstrap containers must exist in airgap registry! (yes/no) [no]: no
```
  Use Airgapped Docker Registry?: If you use an Air-gapped Docker Registry, you will be prompted for more information.
  For a reliable upgrade experience, it is recommended that you must answer yes and take Data Fabric offline during the upgrade process. If the Data Fabric is not offline, upgrade process can get disrupted.
```
Do you want to also take the Data Fabric named <Data Fabric> offline and have it upgraded at this time? (yes/no) [no]: yes
```
  NOTE In certain steps, such as upgrading Kubernetes versions, most of the pods get restarted, including Data Fabric pods like cldb and zk. You must maintain quorum when upgrading zk pods, and this process can take a long time.
  For a reliable upgrade experience, HPE recommends that you answer no and keep Data Fabric offline during the upgrade process. If the Data Fabric is not offline, upgrade process can get disrupted.
```
Would you like the Data Fabric to restart automatically after it is upgraded? Keep it offline if you are going to perform any other major upgrades next such as upgrading Kubernetes. (yes/no) [yes]: no
```
```
We are now ready to upgrade your Kubernetes components...yes
>>> Continue with upgrade? (yes/no) [yes]: yes
```
  Ensure you enter yes.
  
  You can use no if you want to stop the upgrade process now, and continue at another time. For example, you may not want to bring down your Data Fabric at this time.
4. If you are upgrading Data Fabric as part of Hewlett Packard Enterprise upgrade, then skip to Upgrading to HPE Ezmeral Runtime Enterprise 5.6.x. If you are upgrading HPE Ezmeral Data Fabric on Kubernetes, proceed to next step.
5. The bootstrap process takes some time for upgrading the components and creating the respective containers.
  For example:
```
mcs-0    0/1     ContainerCreating   0          10s
```
  When most pods finish creating their containers, the same example looks like:
```
mcs-0    0/1     Running   0          17m
```
  NOTE In the preceding example, Most pods are in Running state, but they are not in Ready state, and therefore the Data Fabric is offline. One exception is the objectstore pod, which is in Init state.
  If the objectstore pod is in Terminating stage as shown in the following example, it will not create a new container:
```
objectstore-zone1-0 0/1 Terminating 5 10h
```
  Delete the pod using following command:
```
kubectl delete pod objectstore-zone1-0 -n <Data Fabric namespace> --force
```
  Ensure that pod is in Init state, and not in Running sate, for example:
```
objectstore-zone1-0 0/1 Init:0/1 0 5s
```
6. If you are using Centos 7, remove the python36 using the following command:
```
yum remove -y python36
```