About
This site contains documentation for HPE Ezmeral Runtime Enterprise, including installation, configuration, administration, and reference content, and information about related solutions. Examples of related solutions include HPE Ezmeral ML Ops and HPE Ezmeral Runtime Analytics for Apache Spark.
5.6 Reference
- HPE Ezmeral Runtime Enterprise 5.6
- Software Versions
- Kubernetes Bundles
  Kubernetes Bundles are software packages that can contain software to support newer Kubernetes versions, updated add-ons, and software fixes. Kubernetes Bundles enable you to update your deployment without requiring you to upgrade to a newer version of HPE Ezmeral Runtime Enterprise.
- Quick Links
- What's New in Version 5.6.x
  This topic summarizes the new features and important changes in HPE Ezmeral Runtime Enterprise 5.6.x compared to HPE Ezmeral Runtime Enterprise 5.5.x.
- Prepackaged Applications
- On-Premises, Hybrid, and Multi-Cloud Deployments
- Third-Party Licenses
- Universal Concepts
- Accessing HPE Ezmeral Runtime Enterprise Applications and Services
- Navigating the GUI
  Describes the screen layout of the HPE Ezmeral Runtime Enterprise graphical user interface (GUI).
- HPE Ezmeral ML Ops
  The topics in this section provide information about machine learning operations (ML Ops/MLOps) using HPE Ezmeral ML Ops in HPE Ezmeral Runtime Enterprise. (Not available with HPE Ezmeral Runtime Enterprise Essentials.)
- Spark on Kubernetes
  The topics in this section provide information about Apache Spark on Kubernetes in HPE Ezmeral Runtime Enterprise. (Not available with HPE Ezmeral Runtime Enterprise Essentials.)
  - Spark Overview
    This topic provides a brief overview of Apache Spark on HPE Ezmeral Runtime Enterprise.
  - Spark Version Comparison Matrix
    This matrix shows the different versions of Spark supported on HPE Ezmeral Runtime Enterprise.
  - Interoperability Matrix for Spark
    This section provides information about support and interoperability for Spark and its components with HPE Ezmeral Runtime Enterprise.
  - Spark Prerequisites
    This topic describes the prerequisites to run Spark Applications on Kubernetes clusters in HPE Ezmeral Runtime Enterprise.
  - Preparing the Spark Environment
    This topic describes how to prepare the environment to run Spark Applications.
  - Spark Support
    This topic describes the Spark enhancements and limitations for HPE Ezmeral Runtime Enterprise.
  - Configuring Memory for Spark Applications
    This topic describes how to set memory options for Spark applications.
  - Spark Images
    This topic lists the images that must be available to install and run Spark Operator, Apache Livy, Spark History Server, Spark Thrift Server, and Hive Metastore. These images enables you to run the Spark applications in an air-gapped environment.
  - Spark Security
    This topic describes the Spark security concepts in HPE Ezmeral Runtime Enterprise.
  - Updating Helm Charts for Spark Services
    This topic describes how to update the Helm charts for Hive Metastore, Livy, Spark History Server, and Spark Thrift Server on HPE Ezmeral Runtime Enterprise.
  - Nvidia Spark-RAPIDS Accelerator for Spark
    This topic describes Nvidia spark-rapids accelerator support for Spark.
  - Submitting and Managing Spark Applications Using HPE Ezmeral Runtime Enterprise new UI
    This section describes how to access HPE Ezmeral Runtime Enterprise new UI to create and monitor Spark applications.
  - Spark Operator
    This topic provides an overview of Spark Operator on HPE Ezmeral Runtime Enterprise.
  - Livy Overview
    This topic provides the overview for Apache Livy on HPE Ezmeral Runtime Enterprise.
  - Submitting Spark Applications Using spark-submit
    This topic describes how to install spark-client Helm chart and submit Spark applications using spark-submit utility in HPE Ezmeral Runtime Enterprise.
  - Delta Lake with Apache Spark
    This section describes the Delta Lake that provides ACID transactions for Apache Spark 3.x.x on HPE Ezmeral Runtime Enterprise.
  - Spark History Server
    This topic provides an overview of Spark History Server.
  - Spark Thrift Server
    This topic provides an overview of Spark Thrift Server.
  - Hive Metastore
    This section describes enhancements to the Hive Metastore for HPE Ezmeral Runtime Enterprise.
  - Using Airflow to Schedule Spark Applications
    This topic describes how to use Airflow to schedule Spark applications on HPE Ezmeral Runtime Enterprise.
  - Creating and Connecting Tenants to HPE Ezmeral Data Fabric on Bare Metal
    This topic describes how to create tenants to connect to HPE Ezmeral Data Fabric on Bare Metal not registered as Tenant Storage.
  - Pulling Images from GCR repository on Local Workstation
    This topic describes how to pull images from GCR repository on your local workstation using minikube single-node environment.
  - (Optional) Connect a Local Workstation
- Kubernetes
- Platform Administration
  Tasks and reference information for Platform Adminstrators (Site Administrators) managing the HPE Ezmeral Runtime Enterprise deployment.
- HPE Ezmeral Data Fabric Introduction
  HPE Ezmeral Data Fabric is a platform for data-driven analytics, ML, and AI workloads. The patented file-system architecture was designed and built for performance, reliability, and scalability. HPE Ezmeral Runtime Enterprise supports multiple implementations of HPE Ezmeral Data Fabric.
- HPE Ezmeral Data Fabric on Kubernetes Administration
  You administer HPE Ezmeral Data Fabric on Kubernetes and Embedded Data Fabric as part of your HPE Ezmeral Runtime Enterprise environment. The external "bare metal" implementation of HPE Ezmeral Data Fabric is administered through its own tools and has its own documentation. (Not available in HPE Ezmeral Runtime Enterprise Essentials.)
- GPU and MIG Support
  This topic provides information about support for NVIDIA GPU and MIG devices on HPE Ezmeral Runtime Enterprise.
- Licensing
- Global Settings
- Planning the Deployment
  A high-level overview of the items to consider when planning an HPE Ezmeral Runtime Enterprise deployment.
- System Requirements
- Deploying the Platform
  The topics in this section describe deploying HPE Ezmeral Runtime Enterprise. Deployment is divided into phases.
- Upgrading to HPE Ezmeral Runtime Enterprise 5.6.x
  This article describes the process to upgrade to the latest 5.6.x version of HPE Ezmeral Runtime Enterprise.
- Upgrading from HPE Ezmeral Runtime Enterprise Essentials
  Upgrade from HPE Ezmeral Runtime Enterprise Essentials to the full-featured HPE Ezmeral Runtime Enterprise or to HPE Ezmeral ML Ops by uploading a license. No additional steps are required.
- Manually Restarting HPE Ezmeral Runtime Enterprise Services
  This topic describes restarting HPE Ezmeral Runtime Enterprise services in non-Kubernetes hosts.
- Uninstalling and Reinstalling HPE Ezmeral Runtime Enterprise
- Support and Troubleshooting
App Workbench 5.1

Spark Support

This topic describes the Spark enhancements and limitations for HPE Ezmeral Runtime Enterprise.

No Support for Apache Spark 3.1.1 Operator

In HPE Ezmeral Runtime Enterprise 5.4, the unified version of Apache Spark 3.x.x (default and recommended) replaces the preview version of Apache Spark 3.1.1 introduced in HPE Ezmeral Runtime Enterprise 5.3.

Hewlett Packard Enterprise no longer supports Apache Spark 3.1.1 and recommends you to move all your Spark applications to Apache Spark 3.x.x.

Spark Limitations

When you configure the Amazon S3 to store event logs, Spark History Server does not show the list of running applications when you click Show incomplete applications. Amazon S3 doesn’t support the append functionality and web UI only displays the list of completed application runs.
Starting from HPE Ezmeral Runtime Enterprise 5.6.0, Spark 3.3.x and later versions support enhanced S3 features introduced in Hadoop 3.x.
Starting from HPE Ezmeral Runtime Enterprise 5.5.0, Spark Operator supports open-source Spark images. You can now build your Spark with Hadoop 3 profile or any other profile of your choice. See Spark Operator for details.
Starting from HPE Ezmeral Runtime Enterprise 5.4.0, Livy does not support ZooKeeper for session recovery. Configure pvc or disabled option for sessionRecovery in all data-fabric (default) tenants or none tenants.

For Spark issues, see Issues and Workarounds.