Apache Livy

The topics in this section provide information about Apache Livy features and limitations in HPE Ezmeral Runtime Enterprise.

HPE Ezmeral Runtime Enterprise supports Apache Livy.

Learn more about supported Livy versions at Interoperability Matrix for Spark.

Apache Livy includes the following new features:

Authentication and Impersonation

Figure 1. Apache Livy Impersonation on HPE Ezmeral Runtime Enterprise


Authentication and impersonation are enabled by default in Livy with the HPE Ezmeral Runtime Enterprise. Like any other container platform service with authentication, Livy authenticates users with LDAP credentials. Livy starts a Livy session with the tenant ticket secrets and specifies proxyUser. Livy server and all Livy sessions run with the same ticket secrets. Livy server provides the user information to the Spark session and Spark session will use that user information to access HPE Ezmeral Data Fabric Filesystem.

For this reason, curl commands need to include the -u "username:password" command string. For example, change this command:
curl -k -v \ 
    -X POST \
    -H "Content-Type: application/json" \
    -d '{}' \
    https://hcp-lb1.qa.lab:10075/sessions
to this:
curl -k -v \ 
    -X POST \
    -H "Content-Type: application/json" \
    -d '{}' \
    -u "username:password" \
    https://hcp-lb1.qa.lab:10075/sessions

Livy User Interface (UI)

The Livy user interface (UI) is secured and accessible via HTTPS from the HPE Ezmeral Runtime Enterprise web interface by navigating to Kubernetes > Tenants > Applications > Service Endpoints. For example:
https://your.co.lab:10046/ui

Spark History Server

The Spark History Server displays all active and completed Livy sessions. Spark History Server web UI is accessible via HTTPS from the HPE Ezmeral Runtime Enterprise web interface by navigating to Kubernetes > Tenants > Applications > Service Endpoints. For example:
https://your.co.lab:10038/
Figure 2. Spark History Server Display of Running and Completed Apps

Livy Session Recovery

Livy allows you to recover and continue working in previous sessions after deleting and restarting the Livy server pod. See Configuring Apache Livy for Session Recovery.
NOTE

You cannot recover the Livy sessions if you delete the tenant. Livy sessions run in the tenant namespace. If you delete the tenant, it will delete the namespace and all the Livy sessions running in that namespace.

Support for Hive Metastore

Livy also supports integration with Hive Metastore. See Configuring Apache Livy for Hive Metastore.