About
This site contains documentation for HPE Ezmeral Runtime Enterprise, including installation, configuration, administration, and reference content, and information about related solutions. Examples of related solutions include HPE Ezmeral ML Ops and HPE Ezmeral Runtime Analytics for Apache Spark.
5.6 Reference
- HPE Ezmeral Runtime Enterprise 5.6
- Software Versions
- Kubernetes Bundles
  Kubernetes Bundles are software packages that can contain software to support newer Kubernetes versions, updated add-ons, and software fixes. Kubernetes Bundles enable you to update your deployment without requiring you to upgrade to a newer version of HPE Ezmeral Runtime Enterprise.
- Quick Links
- What's New in Version 5.6.x
  This topic summarizes the new features and important changes in HPE Ezmeral Runtime Enterprise 5.6.x compared to HPE Ezmeral Runtime Enterprise 5.5.x.
- Prepackaged Applications
- On-Premises, Hybrid, and Multi-Cloud Deployments
- Third-Party Licenses
- Universal Concepts
- Accessing HPE Ezmeral Runtime Enterprise Applications and Services
- Navigating the GUI
  Describes the screen layout of the HPE Ezmeral Runtime Enterprise graphical user interface (GUI).
- HPE Ezmeral ML Ops
  The topics in this section provide information about machine learning operations (ML Ops/MLOps) using HPE Ezmeral ML Ops in HPE Ezmeral Runtime Enterprise. (Not available with HPE Ezmeral Runtime Enterprise Essentials.)
- Spark on Kubernetes
  The topics in this section provide information about Apache Spark on Kubernetes in HPE Ezmeral Runtime Enterprise. (Not available with HPE Ezmeral Runtime Enterprise Essentials.)
  - Spark Overview
    This topic provides a brief overview of Apache Spark on HPE Ezmeral Runtime Enterprise.
  - Spark Version Comparison Matrix
    This matrix shows the different versions of Spark supported on HPE Ezmeral Runtime Enterprise.
  - Interoperability Matrix for Spark
    This section provides information about support and interoperability for Spark and its components with HPE Ezmeral Runtime Enterprise.
  - Spark Prerequisites
    This topic describes the prerequisites to run Spark Applications on Kubernetes clusters in HPE Ezmeral Runtime Enterprise.
  - Preparing the Spark Environment
    This topic describes how to prepare the environment to run Spark Applications.
  - Spark Support
    This topic describes the Spark enhancements and limitations for HPE Ezmeral Runtime Enterprise.
  - Configuring Memory for Spark Applications
    This topic describes how to set memory options for Spark applications.
  - Spark Images
    This topic lists the images that must be available to install and run Spark Operator, Apache Livy, Spark History Server, Spark Thrift Server, and Hive Metastore. These images enables you to run the Spark applications in an air-gapped environment.
  - Spark Security
    This topic describes the Spark security concepts in HPE Ezmeral Runtime Enterprise.
  - Updating Helm Charts for Spark Services
    This topic describes how to update the Helm charts for Hive Metastore, Livy, Spark History Server, and Spark Thrift Server on HPE Ezmeral Runtime Enterprise.
  - Nvidia Spark-RAPIDS Accelerator for Spark
    This topic describes Nvidia spark-rapids accelerator support for Spark.
  - Submitting and Managing Spark Applications Using HPE Ezmeral Runtime Enterprise new UI
    This section describes how to access HPE Ezmeral Runtime Enterprise new UI to create and monitor Spark applications.
  - Spark Operator
    This topic provides an overview of Spark Operator on HPE Ezmeral Runtime Enterprise.
  - Livy Overview
    This topic provides the overview for Apache Livy on HPE Ezmeral Runtime Enterprise.
    - Apache Livy
      The topics in this section provide information about Apache Livy features and limitations in HPE Ezmeral Runtime Enterprise.
    - Configuring Apache Livy for Session Recovery
      Livy supports session recovery on HPE Ezmeral Runtime Enterprise. Even if the Livy server fails, Livy allows you to recover and continue working in previous sessions after deleting and restarting the Livy server pod.
    - Connecting to Livy from a KubeDirector Notebook Application with Spark Magic
      This topic describes how to initiate a Spark session from a Livy endpoint and how to use the %setLivy magic to connect to a different Livy session.
  - Submitting Spark Applications Using spark-submit
    This topic describes how to install spark-client Helm chart and submit Spark applications using spark-submit utility in HPE Ezmeral Runtime Enterprise.
  - Delta Lake with Apache Spark
    This section describes the Delta Lake that provides ACID transactions for Apache Spark 3.x.x on HPE Ezmeral Runtime Enterprise.
  - Spark History Server
    This topic provides an overview of Spark History Server.
  - Spark Thrift Server
    This topic provides an overview of Spark Thrift Server.
  - Hive Metastore
    This section describes enhancements to the Hive Metastore for HPE Ezmeral Runtime Enterprise.
  - Using Airflow to Schedule Spark Applications
    This topic describes how to use Airflow to schedule Spark applications on HPE Ezmeral Runtime Enterprise.
  - Creating and Connecting Tenants to HPE Ezmeral Data Fabric on Bare Metal
    This topic describes how to create tenants to connect to HPE Ezmeral Data Fabric on Bare Metal not registered as Tenant Storage.
  - Pulling Images from GCR repository on Local Workstation
    This topic describes how to pull images from GCR repository on your local workstation using minikube single-node environment.
  - (Optional) Connect a Local Workstation
- Kubernetes
- Platform Administration
  Tasks and reference information for Platform Adminstrators (Site Administrators) managing the HPE Ezmeral Runtime Enterprise deployment.
- HPE Ezmeral Data Fabric Introduction
  HPE Ezmeral Data Fabric is a platform for data-driven analytics, ML, and AI workloads. The patented file-system architecture was designed and built for performance, reliability, and scalability. HPE Ezmeral Runtime Enterprise supports multiple implementations of HPE Ezmeral Data Fabric.
- HPE Ezmeral Data Fabric on Kubernetes Administration
  You administer HPE Ezmeral Data Fabric on Kubernetes and Embedded Data Fabric as part of your HPE Ezmeral Runtime Enterprise environment. The external "bare metal" implementation of HPE Ezmeral Data Fabric is administered through its own tools and has its own documentation. (Not available in HPE Ezmeral Runtime Enterprise Essentials.)
- GPU and MIG Support
  This topic provides information about support for NVIDIA GPU and MIG devices on HPE Ezmeral Runtime Enterprise.
- Licensing
- Global Settings
- Planning the Deployment
  A high-level overview of the items to consider when planning an HPE Ezmeral Runtime Enterprise deployment.
- System Requirements
- Deploying the Platform
  The topics in this section describe deploying HPE Ezmeral Runtime Enterprise. Deployment is divided into phases.
- Upgrading to HPE Ezmeral Runtime Enterprise 5.6.x
  This article describes the process to upgrade to the latest 5.6.x version of HPE Ezmeral Runtime Enterprise.
- Upgrading from HPE Ezmeral Runtime Enterprise Essentials
  Upgrade from HPE Ezmeral Runtime Enterprise Essentials to the full-featured HPE Ezmeral Runtime Enterprise or to HPE Ezmeral ML Ops by uploading a license. No additional steps are required.
- Manually Restarting HPE Ezmeral Runtime Enterprise Services
  This topic describes restarting HPE Ezmeral Runtime Enterprise services in non-Kubernetes hosts.
- Uninstalling and Reinstalling HPE Ezmeral Runtime Enterprise
- Support and Troubleshooting
App Workbench 5.1

Livy Overview

This topic provides the overview for Apache Livy on HPE Ezmeral Runtime Enterprise.

Livy is an HTTP server that allows you to launch Spark applications and submit code statements using the REST API. Livy can be launched as a tenant service in HPE Ezmeral Runtime Enterprise. See Apache Livy Documentation.

Figure 1. Using Apache Livy to Submit Spark Applications

Support for the Apache Livy Operator

HPE Ezmeral Runtime Enterprise integrates Apache Livy. You can only install one version of Livy in one tenant. You cannot simultaneously run Spark 2.x.x and Spark 3.x.x applications in the same tenant using Livy.

Learn more about supported Livy versions at Interoperability Matrix for Spark.

The new Livy features:

Authentication and Impersonation
Livy User Interface (UI)
Spark History Server
Livy Session Recovery
Support for Hive Metastore

NOTE Livy does not support open-source Spark images on HPE Ezmeral Runtime Enterprise.