Upgrading an Existing Image

This article describes how to update an existing application .bin file. This example upgrades an existing CDH 5.7 image to CDH 5.9. This upgrade process does not cover any additional modifications to the image, such as adding services or modifying startscripts. To upgrade an existing image:

  1. Create a new directory to house the image you are going to create, such as /source/<image name>, where /source/<image name> is the name of the new image you are going to create. This example uses /source/cdh59.
  2. Copy the .bin for the existing image to the directory that you created in Step 1.
  3. Navigate to the /opt/bluedata/catalog/bundles directory. This directory contains two subdirectories:
    • /download
    • /install.
  4. If the image you want to modify has already been installed in your App Store, then go to the /install directory. If the image is available in your App Store but has not been installed, then go to the /download directory.
                            [root@yav-100 ~]# cd opt/bluedata/catalog/bundles
                            [root@yav-100 bundles]# ls
                            download install
                            [root@yav-100 bundles]
                            [root@yav-100 downloads]# ls
                            bdcatalog-centos-bluedata-cdh551-1.6.bin        bdcatalog-centos-bluedata-hdp25-ambari-2.0.bin
                            bdcatalog-centos-bluedata-cdh551-edge-1.3.bin   bdcatalog-centos-bluedata-hdp-edge25-2.3.bin
                            bdcatalog-centos-bluedata-cdh57-2.1.bin         bdcatalog-centos-bluedata-mapr510-2.3.bin
                            bdcatalog-centos-bluedata-cdh57-edge-1.1.bin    bdcatalog-centos-bluedata-spark15-2.1.bin
                            bdcatalog-centos-bluedata-hdp23-ambari-1.4.bin  bdcatalog-centos-bluedata-spark16-1.8.bin
                            bdcatalog-centos-bluedata-hdp24-edge-1.0.bin    bdcatalog-centos-bluedata-spark201-1.2.bin
                            bdcatalog-centos-bluedata-hdp24-ambari-1.7.bin  bdcatalog-centos-bluedata-spark201-edge-2.3.bin
  5. Identify the .bin file for your image, and then copy it to the parent source directory that you created in Step 1. This may take several minutes if the image was not installed in your App Store, depending on the file size and transfer speed.
  6. Make the .bin file executable by executing the command chmod +x <bin_name>, where <bin_name> is the name of the .bin file, such as chmod +x bdcatalog-centos-bluedata-cdh57-2.1.bin.
  7. Unpack the .bin file by executing the command ./<bin-name> --payload. This generates two files in the source directory:
    • payload.tar
    • decompress.sh.
                            [root@yav-100 source]# ./bdcatalog-centos-bluedata-cdh57-2.1.bin --payload
                            [root@yav-100 source]# ls
                            bdcatalog-centos-bluedata-cdh57-2.1.bin   cdh59   decompress.sh   payload.tar
  8. Untar the payload.tar file by executing the command tar xvf payload.tar. This creates a new directory, such as bdcatalog-centos-bluedata-cdh57-2.1.
                            [root@yav-100 source]# tar xvf payload.tar
                            bdcatalog-centos-bluedata-cdh57-2.1/
                            bdcatalog-centos-bluedata-cdh57-2.1/bdcatalog-centos-bluedata-cdh57-2.1-src.tgz
                            bdcatalog-centos-bluedata-cdh57-2.1/bluedata-cdh57-centos-2.0.tar.gz
                            bdcatalog-centos-bluedata-cdh57-2.1/cdh5-cm-setup.tgz
                            bdcatalog-centos-bluedata-cdh57-2.1/cdh57CM-centos.json
                            [root@yav-100 source]# ls
                            bdcatalog-centos-bluedata-cdh57-2.1   bdcatalog-centos-bluedata-cdh57-2.1.bin
                            cdh59   decompress.sh   payload.tar
  9. Navigate to the new directory, and untar the <bin name>-src.tgz file by executing the command tar xvzf bdcatalog-centos-bluedata-cdh57-2.1-src.tgz to access the source directory of the .bin file.
                            [root@yav-100 source]# cd bdcatalog-centos-bluedata-cdh57-2.1
                            [root@yav-100 bdcatalog-centos-bluedata-cdh57-2.1]# ls
                            bdcatalog-centos-bluedata-cdh57-2.1-src   bdcatalog-centos-bluedata-cdh57-2.1-src.tgz
                            bdcatalog-cdh57-centos-2.0.tar.gz   cdh57CM-centos.json   cdh5-cm-setup.tgz
                            [root@yav-100 bdcatalog-centos-bluedata-cdh57-2.1]# cd bdcatalog-centos-bluedata-cdh57-2.1-src [root@yav-100 bdcatalog-centos-bluedata-cdh57-2.1-src]# ls
                            cdh57CM-centos.json   cdh57cm.wb   cdh5-cm-setup   image   Logo_Cloudera.png
  10. Copy the contents of the source directory to the new image directory that you created in Step 1. The directory should appear as shown below after the copy is complete:
                            [root@yav-100 source]# cd cdh59
                            [root@yav-100 cdh59]# ls
                            cdh57CM-centos.json   cdh57cm.wb   cdh5-cm-setup   image   Logo_Cloudera.png
  11. If desired, you may clean up your source directory by removing the following items so that only the image directory remains:
    • payload.tar
    • decompress.sh
    • The original .bin file, such as bdcatalog-centos-bluedata-cdh57-2.1.bin.
    • The unpacked .bin file, such as bdcatalog-centos-bluedata-cdh57-2.1.
  12. Change the JSON and .wb file names to reflect the new image version. You do not need to change the logo.png file, nor the image-setup directory (such as version.cdh5-cm-setup) in order to upgrade the image.
                            [root@yav-100 cdh59]# ls
                            cdh59CM-centos.json   cdh59cm.wb   cdh5-cm-setup   image   Logo_Cloudera.png
  13. Modify the .wb file,as needed. This example requires changing the following three items:
    • The catalog load --filepath needs to be updated to reflect the JSON file of the new image, such as changing cdh57CM-centos.json to cdh59CM-centos.json.
    • In the CentOS catalog bundle section, the imgversion needs to be updated. This example demonstrates building a new CDH 5.9 image, so the image version will be 1.0.
    • In the CentOS catalog bundle section, update the catalog save --filepath to also reflect the JSON file of the new image.
  14. Verify that the lines in the RHEL catalog bundle section are commented out, because this example is for a CentOS image.

    The following image shows the changes made to the Dockerfile (changes appear in red text).
                            #
                            # YOUR_ORGANIZATION_NAME must be replaced with a valid organization name. Please
                            # refer to 'help builder organization' for details.
                            #
                            builder organization --name BlueData
                            
                            catalog load --filepath cdh59CM-centos.json
                            appconfig package --dir cdh5-cm-setup
                            logo file --filepath Logo_Cloudera.png
                            
                            #@# Centos catalog bundle
                            image build --basedir image/centos --imgversion 1.0 --os centos
                            catalog save --filepath staging/cdh59CM-centos.json --force
                            sources package --additional cdh5-cm-setup
                            catalogpackage --os=centos
                            
                            ##RHEL catalog bundle
                            #imagebuild --basedir image/rhel --imgversion 2.0 --os rhel
                            #catalogsave --filepath staging/cdh57CM-rhel.json --force
                            #sources package --additional cdh5-cm-setup
                            #catalog package --os=rhel
                            
                            workbench clean
                            ~
                            ~
                            "cdh59cm.wb" 24L, 769C
  15. Navigate to the image/<os> directory (such as centos), which contains the cloudera-manager.repo and Dockerfile. Dockerfile contents will vary by application and configuration. See Sample Docker Files for two sample Dockerfiles.
  16. Open the cloudera-manager.repo file, and then verify that the base URL path is valid for your desired image before you update the repo file. In this example, update the baseurl path to 5.9.0.
                            baseurl=http://archive.coudera.com/cm5/redhat/6/x86_64/cm/5.9.0
                            gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
  17. Navigate to the image/<os> directory, and then open the Dockerfile. The contents of this file will vary depending on the application and its unique configurations. In this example, a total of 15 changes are required. Most of these changes involve simply changing 5.7.0 to 5.9.0). It is recommended that you verify the parcel and rpm names with the distribution (Hadoop, Spark, Cloudera) in order to confirm that all changes required for this upgrade have been made. The following text shows a sample Dockerfile before and after making these changes, where:
    • The before example uses a gray background.
    • The after example uses a black background. Changes are shown in red text.
    # CDH 5.7.0 docker image
    
    
    
                             FROM bluedata/centos6:latest
    
    
    
                             #LABEL Description="This is an image for CDH5.7.0" Vendor="BlueData, Inc"
    
    
    
                             # remove openjdk we installed in base image
    
    
                            RUN yum -y erase java-1.7.0-openjdk
    
    
                            ENV JAVA_HOME ''
    
    
    
                             # Install cloudera manager and parcels
    
    
                            ADD cloudera-manager.repo /etc/yum.repos.d/
    
    
                            RUN yum -y install oracle-j2sdk1.7.x86_64 cloudera-manager-server-db-2-5.7.0   \
    
    
                                                cloudera-manager-server-5.7.0 cloudera-manager-daemons-5.7.0\
    
    
                                                cloudera-manager-agent-5.7.0 krb5*\
    
    
    
                             #Clean up the yum repository
    
    
                            RUN yum clean all; rm -rf /tmp/* /var/tmp/* /var/cache/jum/*
    
    
    
                             # Install python php and install cm-api through pip
    
    
                            RUN wget https://bootstrap.pypa.io/get-pip.py -P /tmp
    
    
                            RUN python /tmp/get-pip.py
    
    
                            RUN pip install cm-api==12.0.0
    
    
    
                             # download parcels for CDH
    
    
                            RUN wget http://archive.cloudera.com/cdhg5/parcels/5.7.0/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel -P /opt/cloudera/parcel-repo
    
    
                            RUN wget http://archive.cloudera.com/cdhg5/parcels/5.7.0/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel.sha1 -P /opt/cloudera/parcel-repo
    
    
                            RUN mv /opt/cloudera/parcel-repo/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.7.0-1.cdh5.7.0.p0.45-e16.parcel.sha
    
    
                            RUN chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
    
    
    
                             ## Download Unlimited JCE policy zip file and copy to the parent directory of
    
    
                            ## this Dockerfile before uncommenting this section
    
    
    
                             #Add UnlimitedJCEPolicyJDK7.zip .
    
    
                            #RUN wzip UnlimtedJCEPolicyJDK7.zip && rm UnlimtedJCEPolicyJDK7.zip
    
    
                            #RUN mv UnlimtedJCEPolicy/US_export_Policy.jar /usr/java/jdk1.7.*-cloudera/jre/lib/security/local_policy.jar
    
    
                            #RUN rm -rf UnlimtedJCEPolicy*
                            # CDH 5.9.0 docker image
                            
                            FROM bluedata/centos6:latest
                            
                            #LABEL Description="This is an image for CDH5.9.0" Vendor="BlueData, Inc"
                            
                            # remove openjdk we installed in base image
                            RUN yum -y erase java-1.7.0-openjdk
                            ENV JAVA_HOME ''
                            
                            # Install cloudera manager and parcels
                            ADD cloudera-manager.repo /etc/yum.repos.d/
                            RUN yum -y install oracle-j2sdk1.7.x86_64 cloudera-manager-server-db-2-5.9.0   \
                                               cloudera-manager-server-5.9.0 cloudera-manager-daemons-5.9.0\
                                               cloudera-manager-agent-5.9.0 krb5*\
                            
                            #Clean up the yum repository
                            RUN yum clean all; rm -rf /tmp/* /var/tmp/* /var/cache/jum/*
                            
                            # Install python php and install cm-api through pip
                            RUN wget https://bootstrap.pypa.io/get-pip.py -P /tmp
                            RUN python /tmp/get-pip.py
                            RUN pip install cm-api==12.0.0
                            
                            # download parcels for CDH
                            RUN wget http://archive.cloudera.com/cdhg5/parcels/5.9.0/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel -P /opt/cloudera/parcel-repo
                            RUN wget http://archive.cloudera.com/cdhg5/parcels/5.9.0/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel.sha1 -P /opt/cloudera/parcel-repo
                            RUN mv /opt/cloudera/parcel-repo/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.9.0-1.cdh5.9.0.p0.23-e16.parcel.sha
                            RUN chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
                            
                            ## Download Unlimited JCE policy zip file and copy to the parent directory of
                            ## this Dockerfile before uncommenting this section
                            
                            #Add UnlimitedJCEPolicyJDK7.zip .
                            #RUN wzip UnlimtedJCEPolicyJDK7.zip && rm UnlimtedJCEPolicyJDK7.zip
                            #RUN mv UnlimtedJCEPolicy/US_export_Policy.jar /usr/java/jdk1.7.*-cloudera/jre/lib/security/local_policy.jar
                            #RUN rm -rf UnlimtedJCEPolicy*
  18. Update the JSON file. To upgrade an image, you will need to make changes to various parameters, as appropriate to the application. This example requires the following changes (In the following examples, the before examples use a gray background, and the after examples use a black background with changes shown in red text):
    • cdh_parcel_repo
    • cdh_full_version
    • cdh_parcel_version
    • source_file
    • label name
    • label description
    • distro_id
    • version
    "catalog_api_version": 2,
    
    
                            "config": {
    
    
                              "config_meta": {     "streaming_jar": "/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar",
    
    
                                "cdh_major_version": "CDH5",
    
    
                                "cdh_parcel_repo": "http://archive.cloudera.com/cdh5/parcels/5.7.0",
    
    
                                "cdh_full_version": "5.7.0",
    
    
                                "cdh_parcel_version": "5.7.0-1.cdh5.7.0.p0.45",
    
    
                                "impala_jar_version": "0.1-SNAPSHOT"
                            "catalog_api_version": 2,
                            "config": {
                              "config_meta": {     "streaming_jar": "/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar",
                                "cdh_major_version": "CDH5",
                                "cdh_parcel_repo": "http://archive.cloudera.com/cdh5/parcels/5.9.1",
                                "cdh_full_version": "5.9.1",
                                "cdh_parcel_version": "5.9.1-1.cdh5.7.1.p0.4",
                                "impala_jar_version": "0.1-SNAPSHOT"
    },
    
    
                            "image": {
    
    
                              "checksum": "31bb8c37ecb491b8a42660a06428eace",
    
    
                              "source_file": "bluedata-cdh57-centos-1.0.tar.gz"
    
    
                            },
    
    
                            "label": {
    
    
                              "name": "CDH 5.7.0 with Cloudera Manager",
    
    
                              "description": "CDH 5.7.0 with MRv1/YARN and HBase support. Includes Pig, Hive, Hue and Spark."
    
    
                            },
    
    
                            "distro_id": "bluedata/cdh57",
    
    
                            "version": "2.1",
    
    
                            "services": [
    
    
                              {
                            },
                            "image": {
                              "checksum": "31bb8c37ecb491b8a42660a06428eace",
                              "source_file": "bluedata-cdh59-centos-1.0.tar.gz"
                            },
                            "label": {
                              "name": "CDH 5.9.1 with Cloudera Manager",
                              "description": "CDH 5.9.1 with MRv1/YARN and HBase support. Includes Pig, Hive, Hue and Spark."
                            },
                            "distro_id": "bluedata/cdh59",
                            "version": "1.0",
                            "services": [
                              {
  19. Execute the .wb by executing the command ./<wb name>.wb, such as ./cdh59cm.wb. Image creation can take several minutes depending on the size and complexity of your image. You will be alerted if there are any errors. You will also be alerted once the build is successful.
  20. Once the bin build process is completed, copy your new image .bin from the /deliverables directory that is generated during the build.
                            Successfully built bluedata/cdhg59-centos:1.0.
                            
                            Saving bluedata/cd59-centos:1.0 as /root/source/cdh59/staging/bluedata-cdh-59-centos-1.0.tar.gz
                            Packaging the entry for centos.
                            Catalog bundle is saved at /root/source/cdh59/deliverables/bdcatalog-centos-bluedata-cdh-59-2.1.bin
  21. Copy the new .bin to </srv/bluedata/catalog, and then make it executable by executing the command chmod +x <bin name>.
  22. Log into the HPE Ezmeral Runtime Enterprise web interface as a Platform Administrator, navigate to the App Store screen, and then click Refresh.

    The new image will appear in the App Store with a red New banner across the tile.

  23. Click the Install button to install the image, and then create a test cluster as a final validation step.