Upgrading an Existing Image

This article describes how to update an existing application .bin file. This example upgrades an existing CDH 5.7 image to CDH 5.9. This upgrade process does not cover any additional modifications to the image, such as adding services or modifying startscripts. To upgrade an existing image:

Create a new directory to house the image you are going to create, such as /source/<image name>, where /source/<image name> is the name of the new image you are going to create. This example uses /source/cdh59.
Copy the .bin for the existing image to the directory that you created in Step 1.
Navigate to the /opt/bluedata/catalog/bundles directory. This directory contains two subdirectories:
- /download
- /install.

If the image you want to modify has already been installed in your App Store, then go to the /install directory. If the image is available in your App Store but has not been installed, then go to the /download directory.

                        [root@yav-100 ~]# cd opt/bluedata/catalog/bundles
                        [root@yav-100 bundles]# ls
                        download install
                        [root@yav-100 bundles]
                        [root@yav-100 downloads]# ls
                        bdcatalog-centos-bluedata-cdh551-1.6.bin        bdcatalog-centos-bluedata-hdp25-ambari-2.0.bin
                        bdcatalog-centos-bluedata-cdh551-edge-1.3.bin   bdcatalog-centos-bluedata-hdp-edge25-2.3.bin
                        bdcatalog-centos-bluedata-cdh57-2.1.bin         bdcatalog-centos-bluedata-mapr510-2.3.bin
                        bdcatalog-centos-bluedata-cdh57-edge-1.1.bin    bdcatalog-centos-bluedata-spark15-2.1.bin
                        bdcatalog-centos-bluedata-hdp23-ambari-1.4.bin  bdcatalog-centos-bluedata-spark16-1.8.bin
                        bdcatalog-centos-bluedata-hdp24-edge-1.0.bin    bdcatalog-centos-bluedata-spark201-1.2.bin
                        bdcatalog-centos-bluedata-hdp24-ambari-1.7.bin  bdcatalog-centos-bluedata-spark201-edge-2.3.bin

Identify the .bin file for your image, and then copy it to the parent source directory that you created in Step 1. This may take several minutes if the image was not installed in your App Store, depending on the file size and transfer speed.
Make the .bin file executable by executing the command chmod +x <bin_name>, where <bin_name> is the name of the .bin file, such as chmod +x bdcatalog-centos-bluedata-cdh57-2.1.bin.

Unpack the .bin file by executing the command ./<bin-name> --payload. This generates two files in the source directory:

payload.tar
decompress.sh.

                        [root@yav-100 source]# ./bdcatalog-centos-bluedata-cdh57-2.1.bin --payload
                        [root@yav-100 source]# ls
                        bdcatalog-centos-bluedata-cdh57-2.1.bin   cdh59   decompress.sh   payload.tar

Untar the payload.tar file by executing the command tar xvf payload.tar. This creates a new directory, such as bdcatalog-centos-bluedata-cdh57-2.1.

                        [root@yav-100 source]# tar xvf payload.tar
                        bdcatalog-centos-bluedata-cdh57-2.1/
                        bdcatalog-centos-bluedata-cdh57-2.1/bdcatalog-centos-bluedata-cdh57-2.1-src.tgz
                        bdcatalog-centos-bluedata-cdh57-2.1/bluedata-cdh57-centos-2.0.tar.gz
                        bdcatalog-centos-bluedata-cdh57-2.1/cdh5-cm-setup.tgz
                        bdcatalog-centos-bluedata-cdh57-2.1/cdh57CM-centos.json
                        [root@yav-100 source]# ls
                        bdcatalog-centos-bluedata-cdh57-2.1   bdcatalog-centos-bluedata-cdh57-2.1.bin
                        cdh59   decompress.sh   payload.tar

Navigate to the new directory, and untar the

<bin
                    name>-src.tgz

file by executing the command

tar xvzf
                    bdcatalog-centos-bluedata-cdh57-2.1-src.tgz

to access the source directory of the .bin file.

                        [root@yav-100 source]# cd bdcatalog-centos-bluedata-cdh57-2.1
                        [root@yav-100 bdcatalog-centos-bluedata-cdh57-2.1]# ls
                        bdcatalog-centos-bluedata-cdh57-2.1-src   bdcatalog-centos-bluedata-cdh57-2.1-src.tgz
                        bdcatalog-cdh57-centos-2.0.tar.gz   cdh57CM-centos.json   cdh5-cm-setup.tgz
                        [root@yav-100 bdcatalog-centos-bluedata-cdh57-2.1]# cd bdcatalog-centos-bluedata-cdh57-2.1-src [root@yav-100 bdcatalog-centos-bluedata-cdh57-2.1-src]# ls
                        cdh57CM-centos.json   cdh57cm.wb   cdh5-cm-setup   image   Logo_Cloudera.png

Copy the contents of the source directory to the new image directory that you created in Step 1. The directory should appear as shown below after the copy is complete:

                        [root@yav-100 source]# cd cdh59
                        [root@yav-100 cdh59]# ls
                        cdh57CM-centos.json   cdh57cm.wb   cdh5-cm-setup   image   Logo_Cloudera.png

If desired, you may clean up your source directory by removing the following items so that only the image directory remains:
- payload.tar
- decompress.sh
- The original .bin file, such as bdcatalog-centos-bluedata-cdh57-2.1.bin.
- The unpacked .bin file, such as bdcatalog-centos-bluedata-cdh57-2.1.
Change the JSON and .wb file names to reflect the new image version. You do not need to change the logo.png file, nor the image-setup directory (such as version.cdh5-cm-setup) in order to upgrade the image.
```
                        [root@yav-100 cdh59]# ls
                        cdh59CM-centos.json   cdh59cm.wb   cdh5-cm-setup   image   Logo_Cloudera.png
```
Modify the .wb file,as needed. This example requires changing the following three items:
- The catalog load --filepath needs to be updated to reflect the JSON file of the new image, such as changing cdh57CM-centos.json to cdh59CM-centos.json.
- In the CentOS catalog bundle section, the imgversion needs to be updated. This example demonstrates building a new CDH 5.9 image, so the image version will be 1.0.
- In the CentOS catalog bundle section, update the catalog save --filepath to also reflect the JSON file of the new image.

Verify that the lines in the RHEL catalog bundle section are commented out, because this example is for a CentOS image.

The following image shows the changes made to the Dockerfile (changes appear in red text).

                        #
                        # YOUR_ORGANIZATION_NAME must be replaced with a valid organization name. Please
                        # refer to 'help builder organization' for details.
                        #
                        builder organization --name BlueData
                        
                        catalog load --filepath cdh59CM-centos.json
                        appconfig package --dir cdh5-cm-setup
                        logo file --filepath Logo_Cloudera.png
                        
                        #@# Centos catalog bundle
                        image build --basedir image/centos --imgversion 1.0 --os centos
                        catalog save --filepath staging/cdh59CM-centos.json --force
                        sources package --additional cdh5-cm-setup
                        catalogpackage --os=centos
                        
                        ##RHEL catalog bundle
                        #imagebuild --basedir image/rhel --imgversion 2.0 --os rhel
                        #catalogsave --filepath staging/cdh57CM-rhel.json --force
                        #sources package --additional cdh5-cm-setup
                        #catalog package --os=rhel
                        
                        workbench clean
                        ~
                        ~
                        "cdh59cm.wb" 24L, 769C

Navigate to the image/<os> directory (such as centos), which contains the cloudera-manager.repo and Dockerfile. Dockerfile contents will vary by application and configuration. See Sample Docker Files for two sample Dockerfiles.

Open the cloudera-manager.repo file, and then verify that the base URL path is valid for your desired image before you update the repo file. In this example, update the baseurl path to 5.9.0.

                        baseurl=http://archive.coudera.com/cm5/redhat/6/x86_64/cm/5.9.0
                        gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera

Navigate to the image/<os> directory, and then open the Dockerfile. The contents of this file will vary depending on the application and its unique configurations. In this example, a total of 15 changes are required. Most of these changes involve simply changing 5.7.0 to 5.9.0). It is recommended that you verify the parcel and rpm names with the distribution (Hadoop, Spark, Cloudera) in order to confirm that all changes required for this upgrade have been made. The following text shows a sample Dockerfile before and after making these changes, where:

The before example uses a gray background.
The after example uses a black background. Changes are shown in red text.

# CDH 5.7.0 docker image



                         FROM bluedata/centos6:latest



                         #LABEL Description="This is an image for CDH5.7.0" Vendor="BlueData, Inc"



                         # remove openjdk we installed in base image


                        RUN yum -y erase java-1.7.0-openjdk


                        ENV JAVA_HOME ''



                         # Install cloudera manager and parcels


                        ADD cloudera-manager.repo /etc/yum.repos.d/


                        RUN yum -y install oracle-j2sdk1.7.x86_64 cloudera-manager-server-db-2-5.7.0   \


                                            cloudera-manager-server-5.7.0 cloudera-manager-daemons-5.7.0\


                                            cloudera-manager-agent-5.7.0 krb5*\



                         #Clean up the yum repository


                        RUN yum clean all; rm -rf /tmp/* /var/tmp/* /var/cache/jum/*



                         # Install python php and install cm-api through pip


                        RUN wget https://bootstrap.pypa.io/get-pip.py -P /tmp


                        RUN python /tmp/get-pip.py


                        RUN pip install cm-api==12.0.0



                         # download parcels for CDH


                        RUN wget http://archive.cloudera.com/cdhg5/parcels/5.7.0/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel -P /opt/cloudera/parcel-repo


                        RUN wget http://archive.cloudera.com/cdhg5/parcels/5.7.0/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel.sha1 -P /opt/cloudera/parcel-repo


                        RUN mv /opt/cloudera/parcel-repo/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel.sha


                        RUN chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*



                         ## Download Unlimited JCE policy zip file and copy to the parent directory of


                        ## this Dockerfile before uncommenting this section



                         #Add UnlimitedJCEPolicyJDK7.zip .


                        #RUN wzip UnlimtedJCEPolicyJDK7.zip && rm UnlimtedJCEPolicyJDK7.zip


                        #RUN mv UnlimtedJCEPolicy/US_export_Policy.jar /usr/java/jdk1.7.*-cloudera/jre/lib/security/local_policy.jar


                        #RUN rm -rf UnlimtedJCEPolicy*

                        # CDH 5.9.0 docker image
                        
                        FROM bluedata/centos6:latest
                        
                        #LABEL Description="This is an image for CDH5.9.0" Vendor="BlueData, Inc"
                        
                        # remove openjdk we installed in base image
                        RUN yum -y erase java-1.7.0-openjdk
                        ENV JAVA_HOME ''
                        
                        # Install cloudera manager and parcels
                        ADD cloudera-manager.repo /etc/yum.repos.d/
                        RUN yum -y install oracle-j2sdk1.7.x86_64 cloudera-manager-server-db-2-5.9.0   \
                                           cloudera-manager-server-5.9.0 cloudera-manager-daemons-5.9.0\
                                           cloudera-manager-agent-5.9.0 krb5*\
                        
                        #Clean up the yum repository
                        RUN yum clean all; rm -rf /tmp/* /var/tmp/* /var/cache/jum/*
                        
                        # Install python php and install cm-api through pip
                        RUN wget https://bootstrap.pypa.io/get-pip.py -P /tmp
                        RUN python /tmp/get-pip.py
                        RUN pip install cm-api==12.0.0
                        
                        # download parcels for CDH
                        RUN wget http://archive.cloudera.com/cdhg5/parcels/5.9.0/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel -P /opt/cloudera/parcel-repo
                        RUN wget http://archive.cloudera.com/cdhg5/parcels/5.9.0/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel.sha1 -P /opt/cloudera/parcel-repo
                        RUN mv /opt/cloudera/parcel-repo/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel.sha
                        RUN chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
                        
                        ## Download Unlimited JCE policy zip file and copy to the parent directory of
                        ## this Dockerfile before uncommenting this section
                        
                        #Add UnlimitedJCEPolicyJDK7.zip .
                        #RUN wzip UnlimtedJCEPolicyJDK7.zip && rm UnlimtedJCEPolicyJDK7.zip
                        #RUN mv UnlimtedJCEPolicy/US_export_Policy.jar /usr/java/jdk1.7.*-cloudera/jre/lib/security/local_policy.jar
                        #RUN rm -rf UnlimtedJCEPolicy*

Update the JSON file. To upgrade an image, you will need to make changes to various parameters, as appropriate to the application. This example requires the following changes (In the following examples, the before examples use a gray background, and the after examples use a black background with changes shown in red text):

cdh_parcel_repo
cdh_full_version
cdh_parcel_version
source_file
label name
label description
distro_id
version

"catalog_api_version": 2,


                        "config": {


                          "config_meta": {     "streaming_jar": "/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar",


                            "cdh_major_version": "CDH5",


                            "cdh_parcel_repo": "http://archive.cloudera.com/cdh5/parcels/5.7.0",


                            "cdh_full_version": "5.7.0",


                            "cdh_parcel_version": "5.7.0-1.cdh5.7.0.p0.45",


                            "impala_jar_version": "0.1-SNAPSHOT"

                        "catalog_api_version": 2,
                        "config": {
                          "config_meta": {     "streaming_jar": "/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar",
                            "cdh_major_version": "CDH5",
                            "cdh_parcel_repo": "http://archive.cloudera.com/cdh5/parcels/5.9.1",
                            "cdh_full_version": "5.9.1",
                            "cdh_parcel_version": "5.9.1-1.cdh5.7.1.p0.4",
                            "impala_jar_version": "0.1-SNAPSHOT"

},


                        "image": {


                          "checksum": "31bb8c37ecb491b8a42660a06428eace",


                          "source_file": "bluedata-cdh57-centos-1.0.tar.gz"


                        },


                        "label": {


                          "name": "CDH 5.7.0 with Cloudera Manager",


                          "description": "CDH 5.7.0 with MRv1/YARN and HBase support. Includes Pig, Hive, Hue and Spark."


                        },


                        "distro_id": "bluedata/cdh57",


                        "version": "2.1",


                        "services": [


                          {

                        },
                        "image": {
                          "checksum": "31bb8c37ecb491b8a42660a06428eace",
                          "source_file": "bluedata-cdh59-centos-1.0.tar.gz"
                        },
                        "label": {
                          "name": "CDH 5.9.1 with Cloudera Manager",
                          "description": "CDH 5.9.1 with MRv1/YARN and HBase support. Includes Pig, Hive, Hue and Spark."
                        },
                        "distro_id": "bluedata/cdh59",
                        "version": "1.0",
                        "services": [
                          {

Execute the .wb by executing the command ./<wb name>.wb, such as ./cdh59cm.wb. Image creation can take several minutes depending on the size and complexity of your image. You will be alerted if there are any errors. You will also be alerted once the build is successful.

Once the bin build process is completed, copy your new image .bin from the /deliverables directory that is generated during the build.

                        Successfully built bluedata/cdhg59-centos:1.0.
                        
                        Saving bluedata/cd59-centos:1.0 as /root/source/cdh59/staging/bluedata-cdh-59-centos-1.0.tar.gz
                        Packaging the entry for centos.
                        Catalog bundle is saved at /root/source/cdh59/deliverables/bdcatalog-centos-bluedata-cdh-59-2.1.bin

Copy the new .bin to </srv/bluedata/catalog, and then make it executable by executing the command chmod +x <bin name>.
Log into the HPE Ezmeral Runtime Enterprise web interface as a Platform Administrator, navigate to the App Store screen, and then click Refresh.

The new image will appear in the App Store with a red New banner across the tile.
Click the Install button to install the image, and then create a test cluster as a final validation step.

HPE Ezmeral Runtime Enterprise 5.6 Documentation
Abstract	HPE Ezmeral Container Platform is a unified container platform built on open source Kubernetes and designed for both cloud-native applications and non-cloud-native applications running on any infrastructure either on-premises, in multiple public clouds, in a hybrid model, or at the edge.
Published	October 2024
Edition	5.6.0