Installing and Configuring Apache Livy
This section describes how to install and configure Apache Livy on HPE Ezmeral Runtime Enterprise.
Prerequisites
- Log in as a Kubernetes Tenant Administrator or a Kubernetes Tenant Member in HPE Ezmeral Runtime Enterprise.
About this task
In HPE Ezmeral Runtime Enterprise, install Livy for Apache Spark using the GUI or manually using the Helm chart.
You can only install one version of Livy in one tenant. You cannot simultaneously run Spark 2.x.x and Spark 3.x.x applications in the same tenant using Livy.
Learn more about supported Livy versions at Interoperability Matrix for Spark.
Installing Apache Livy Using the GUI
About this task
Procedure
- Log in as a Kubernetes Tenant Administrator or a Kubernetes Tenant Member on the HPE Ezmeral Runtime Enterprise GUI.
- Click Applications in the main menu. You will see Kubernetes Applications tiles under KubeDirector tab.
- Navigate to Livy tile and click Launch.
-
Configure Cluster Detail and
Settings on Create
Application screen.
- Cluster Detail:
- Enter the Name and Description of the application.
- Settings:
- Set the CPU and Memory (GB) resources.
-
To view
yaml
, click Edit/Launch yaml. - Click Submit.
Results
Installing Apache Livy Using the Helm
Prerequisites
- Install and configure Helm 3.
- Install and configure kubectl.
About this task
Install the Apache Livy on Data Fabric tenants which are HPE Ezmeral Data Fabric on Kubernetes tenants or HPE Ezmeral Data Fabric on Bare Metal tenants or non Data Fabric tenants using the Helm Chart.
See Livy Helm Chart.
Procedure
- Installing Apache Livy on Data Fabric
tenants:
-
To
helm install
the Apache Livy for Apache Spark 3.x.x, run the following command:helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace>
-
To
helm install
the Apache Livy for Apache Spark 2.x.x, run the following command:helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace> \ --set image.imageName=<livy-image-name> \ --set image.tag=<livy-image-tag> \ --set livyVersion=<livy-version> \ --set deImage=<spark-version>:<spark-image-name>
-
- Installing Apache Livy on non Data Fabric
tenants:
-
To
helm install
the Apache Livy for Apache Spark 3.x.x, run the following command:helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace> --set tenantIsUnsecure=true
-
To
helm install
the Apache Livy for Apache Spark 2.x.x, run the following command:helm install <livy-name> ./<path-to-livy-chart> -n <tenant-namespace> --set tenantIsUnsecure=true \ --set image.imageName=<livy-image-name> \ --set image.tag=<livy-image-tag> \ --set livyVersion=<livy-version> \ --set deImage=<spark-version>:<spark-image-name>
-
-
Installing the Livy Helm chart in a non-tenant namespace can cause error due to missing configmaps and secrets.
- If you are using PVC, ensure that PVC is configured in the same tenant namespace as a Livy namespace.
- Configure Livy to work with Spark History Server and Hive
Metastore on
values.yaml
file.
Running the helm install
installs Apache Livy for Apache
Spark in a tenant namespace.
Example
For HPE Ezmeral Runtime Enterprise 5.4.0:
- Installing Apache Livy on Data Fabric tenants:
- To
helm install
the Apache Livy 0.7 for Apache Spark 3.1.2, run the following command:helm install livy ./livy-chart -n sampletenant
-
To
helm install
the Apache Livy 0.5 for Apache Spark 2.4.7, run the following command:helm install livy ./livy-chart -n sampletenant \ --set image.imageName=livy-0.5.0 \ --set image.tag=202112061039R \ --set livyVersion=0.5.0 \ --set deImage=spark-2.4.7:202112061039R
- To
- Installing Apache Livy on non Data Fabric tenants:
- To
helm install
the Apache Livy 0.7 for Apache Spark 3.1.2, run the following command:helm install livy ./livy-chart -n sampletenant --set tenantIsUnsecure=true
-
To
helm install
the Apache Livy 0.5 for Apache Spark 2.4.7, run the following command:helm install livy ./livy-chart -n sampletenant --set tenantIsUnsecure=true \ --set image.imageName=livy-0.5.0 \ --set image.tag=202112061039R \ --set livyVersion=0.5.0 \ --set deImage=spark-2.4.7:202112061039R
- To