Installing and Configuring Apache Livy

This section describes how to install and configure Apache Livy on HPE Ezmeral Runtime Enterprise.

Prerequisites

  1. Log in as a Kubernetes Tenant Administrator or a Kubernetes Tenant Member in HPE Ezmeral Runtime Enterprise.

About this task

In HPE Ezmeral Runtime Enterprise, install Livy for Apache Spark using the GUI or manually using the Helm chart.

You can only install one version of Livy in one tenant. You cannot simultaneously run Spark 2.x.x and Spark 3.x.x applications in the same tenant using Livy.

Learn more about supported Livy versions at Interoperability Matrix for Spark.

Installing Apache Livy Using the GUI

About this task

Install the Apache Livy for Apache Spark by using the HPE Ezmeral Runtime Enterprise GUI.

Procedure

  1. Log in as a Kubernetes Tenant Administrator or a Kubernetes Tenant Member on the HPE Ezmeral Runtime Enterprise GUI.
  2. Click Applications in the main menu. You will see Kubernetes Applications tiles under KubeDirector tab.
  3. Navigate to Livy tile and click Launch.
  4. Configure Cluster Detail and Settings on Create Application screen.
    Cluster Detail:
    Enter the Name and Description of the application.
    Settings:
    Set the CPU and Memory (GB) resources.
    To connect Livy with Hive Metastore, enter ConfigMap created by Hive Metastore in Hive Metastore Source.
    To get the ConfigMap name, run the following command:
    kubectl get cm -n <tenant-namespace> | grep hive
    To enable Session Recovery Settings, check Session Recovery and select Session Recovery Storage.
    To set Spark History Server Settings, check Spark History Server and set PVC Name and Event Log Directory.
    To get the PVC Name, run the following command:
    kubectl get pods <shs-name> -n <tenant-namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}

    To get the Event Log Directory, access Spark History Server web UI from HPE Ezmeral Runtime Enterprise web interface by navigating to Applications> Service Endpoints.

    In HPE Ezmeral Runtime Enterprise 5.4.0, to set Air Gap Settings, check Air Gap and set Base Repository, Image, Image Tag, Image Pull Secret. See Spark Images.
    NOTE
    To set the additional configuration options not available through HPE Ezmeral Runtime Enterprise GUI , edit the values.yaml file and install Livy using the helm charts as described in Installing Apache Livy Using the Helm section.
  5. To view yaml, click Edit/Launch yaml.
  6. Click Submit.

Results

The GUI installs the Apache Livy for Apache Spark in a tenant namespace.

Installing Apache Livy Using the Helm

Prerequisites

  1. Install and configure Helm 3.
  2. Install and configure kubectl.

About this task

Install the Apache Livy on Data Fabric tenants which are HPE Ezmeral Data Fabric on Kubernetes tenants or HPE Ezmeral Data Fabric on Bare Metal tenants or non Data Fabric tenants using the Helm Chart.

Procedure

Helm install the Apache Livy on HPE Ezmeral Runtime Enterprise:
  • Installing Apache Livy on Data Fabric tenants:
    • To helm install the Apache Livy for Apache Spark 3.x.x, run the following command:
      helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace>
    • To helm install the Apache Livy for Apache Spark 2.x.x, run the following command:
      helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace> \
--set image.imageName=<livy-image-name> \
--set image.tag=<livy-image-tag> \
--set livyVersion=<livy-version> \
--set deImage=<spark-version>:<spark-image-name>
  • Installing Apache Livy on non Data Fabric tenants:
    • To helm install the Apache Livy for Apache Spark 3.x.x, run the following command:
      helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace> --set tenantIsUnsecure=true
    • To helm install the Apache Livy for Apache Spark 2.x.x, run the following command:
      helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace> --set tenantIsUnsecure=true \
--set image.imageName=<livy-image-name> \
--set image.tag=<livy-image-tag> \
--set livyVersion=<livy-version> \
--set deImage=<spark-version>:<spark-image-name>

      See Setting Custom KeyStore.

NOTE

  • Installing the Livy Helm chart in a non-tenant namespace can cause error due to missing configmaps and secrets.

  • If you are using PVC, ensure that PVC is configured in the same tenant namespace as a Livy namespace.
  • Configure Livy to work with Spark History Server and Hive Metastore on values.yaml file.

Running the helm install installs Apache Livy for Apache Spark in a tenant namespace.

Example

For HPE Ezmeral Runtime Enterprise 5.4.0:

  • Installing Apache Livy on Data Fabric tenants:
    • To helm install the Apache Livy 0.7 for Apache Spark 3.1.2, run the following command:
      helm install livy ./livy-chart -n sampletenant
    • To helm install the Apache Livy 0.5 for Apache Spark 2.4.7, run the following command:
      helm install livy ./livy-chart -n sampletenant \
--set image.imageName=livy-0.5.0 \
--set image.tag=202112061039R \
--set livyVersion=0.5.0 \
--set deImage=spark-2.4.7:202112061039R
  • Installing Apache Livy on non Data Fabric tenants:
    • To helm install the Apache Livy 0.7 for Apache Spark 3.1.2, run the following command:
      helm install livy ./livy-chart -n sampletenant --set tenantIsUnsecure=true
    • To helm install the Apache Livy 0.5 for Apache Spark 2.4.7, run the following command:
      helm install livy ./livy-chart -n sampletenant --set tenantIsUnsecure=true \
--set image.imageName=livy-0.5.0 \
--set image.tag=202112061039R \
--set livyVersion=0.5.0 \
--set deImage=spark-2.4.7:202112061039R
NOTE
You can modify the above examples based on your Livy, Spark, and HPE Ezmeral Runtime Enterprise versions for successful installation of Livy. See Interoperability Matrix for Spark.