About
This site contains documentation for HPE Ezmeral Runtime Enterprise, including installation, configuration, administration, and reference content, and information about related solutions. Examples of related solutions include HPE Ezmeral ML Ops and HPE Ezmeral Runtime Analytics for Apache Spark.
5.6 Reference
- HPE Ezmeral Runtime Enterprise 5.6
- Software Versions
- Kubernetes Bundles
  Kubernetes Bundles are software packages that can contain software to support newer Kubernetes versions, updated add-ons, and software fixes. Kubernetes Bundles enable you to update your deployment without requiring you to upgrade to a newer version of HPE Ezmeral Runtime Enterprise.
- Quick Links
- What's New in Version 5.6.x
  This topic summarizes the new features and important changes in HPE Ezmeral Runtime Enterprise 5.6.x compared to HPE Ezmeral Runtime Enterprise 5.5.x.
- Prepackaged Applications
- On-Premises, Hybrid, and Multi-Cloud Deployments
- Third-Party Licenses
- Universal Concepts
- Accessing HPE Ezmeral Runtime Enterprise Applications and Services
- Navigating the GUI
  Describes the screen layout of the HPE Ezmeral Runtime Enterprise graphical user interface (GUI).
- HPE Ezmeral ML Ops
  The topics in this section provide information about machine learning operations (ML Ops/MLOps) using HPE Ezmeral ML Ops in HPE Ezmeral Runtime Enterprise. (Not available with HPE Ezmeral Runtime Enterprise Essentials.)
- Spark on Kubernetes
  The topics in this section provide information about Apache Spark on Kubernetes in HPE Ezmeral Runtime Enterprise. (Not available with HPE Ezmeral Runtime Enterprise Essentials.)
  - Spark Overview
    This topic provides a brief overview of Apache Spark on HPE Ezmeral Runtime Enterprise.
  - Spark Version Comparison Matrix
    This matrix shows the different versions of Spark supported on HPE Ezmeral Runtime Enterprise.
  - Interoperability Matrix for Spark
    This section provides information about support and interoperability for Spark and its components with HPE Ezmeral Runtime Enterprise.
  - Spark Prerequisites
    This topic describes the prerequisites to run Spark Applications on Kubernetes clusters in HPE Ezmeral Runtime Enterprise.
  - Preparing the Spark Environment
    This topic describes how to prepare the environment to run Spark Applications.
  - Spark Support
    This topic describes the Spark enhancements and limitations for HPE Ezmeral Runtime Enterprise.
  - Configuring Memory for Spark Applications
    This topic describes how to set memory options for Spark applications.
  - Spark Images
    This topic lists the images that must be available to install and run Spark Operator, Apache Livy, Spark History Server, Spark Thrift Server, and Hive Metastore. These images enables you to run the Spark applications in an air-gapped environment.
  - Spark Security
    This topic describes the Spark security concepts in HPE Ezmeral Runtime Enterprise.
  - Updating Helm Charts for Spark Services
    This topic describes how to update the Helm charts for Hive Metastore, Livy, Spark History Server, and Spark Thrift Server on HPE Ezmeral Runtime Enterprise.
  - Nvidia Spark-RAPIDS Accelerator for Spark
    This topic describes Nvidia spark-rapids accelerator support for Spark.
  - Submitting and Managing Spark Applications Using HPE Ezmeral Runtime Enterprise new UI
    This section describes how to access HPE Ezmeral Runtime Enterprise new UI to create and monitor Spark applications.
  - Spark Operator
    This topic provides an overview of Spark Operator on HPE Ezmeral Runtime Enterprise.
  - Livy Overview
    This topic provides the overview for Apache Livy on HPE Ezmeral Runtime Enterprise.
  - Submitting Spark Applications Using spark-submit
    This topic describes how to install spark-client Helm chart and submit Spark applications using spark-submit utility in HPE Ezmeral Runtime Enterprise.
  - Delta Lake with Apache Spark
    This section describes the Delta Lake that provides ACID transactions for Apache Spark 3.x.x on HPE Ezmeral Runtime Enterprise.
  - Spark History Server
    This topic provides an overview of Spark History Server.
    - Installing and Configuring Spark History Server
      This section describes how to install and configure Spark History Server on HPE Ezmeral Runtime Enterprise.
    - Using Custom KeyStore
      This topic describes how to use custom KeyStore for Spark History Server SSL encryption for non data-fabric (none) tenants.
    - Configuring Spark Applications to Write and View Logs
      This section guides you through configuring your Spark Application CRs to write logs in the event directory and view the Spark Application details in Spark web UI.
    - Configuring Resource Limits on Spark History Server
      This section guides you through configuring resource limits for Spark History Server on ResourceQuota configured namespace.
    - Using Amazon S3 to Store Logs
    - Deleting Spark History Server
      This section describes how to delete or uninstall Spark History Server from HPE Ezmeral Runtime Enterprise.
  - Spark Thrift Server
    This topic provides an overview of Spark Thrift Server.
  - Hive Metastore
    This section describes enhancements to the Hive Metastore for HPE Ezmeral Runtime Enterprise.
  - Using Airflow to Schedule Spark Applications
    This topic describes how to use Airflow to schedule Spark applications on HPE Ezmeral Runtime Enterprise.
  - Creating and Connecting Tenants to HPE Ezmeral Data Fabric on Bare Metal
    This topic describes how to create tenants to connect to HPE Ezmeral Data Fabric on Bare Metal not registered as Tenant Storage.
  - Pulling Images from GCR repository on Local Workstation
    This topic describes how to pull images from GCR repository on your local workstation using minikube single-node environment.
  - (Optional) Connect a Local Workstation
- Kubernetes
- Platform Administration
  Tasks and reference information for Platform Adminstrators (Site Administrators) managing the HPE Ezmeral Runtime Enterprise deployment.
- HPE Ezmeral Data Fabric Introduction
  HPE Ezmeral Data Fabric is a platform for data-driven analytics, ML, and AI workloads. The patented file-system architecture was designed and built for performance, reliability, and scalability. HPE Ezmeral Runtime Enterprise supports multiple implementations of HPE Ezmeral Data Fabric.
- HPE Ezmeral Data Fabric on Kubernetes Administration
  You administer HPE Ezmeral Data Fabric on Kubernetes and Embedded Data Fabric as part of your HPE Ezmeral Runtime Enterprise environment. The external "bare metal" implementation of HPE Ezmeral Data Fabric is administered through its own tools and has its own documentation. (Not available in HPE Ezmeral Runtime Enterprise Essentials.)
- GPU and MIG Support
  This topic provides information about support for NVIDIA GPU and MIG devices on HPE Ezmeral Runtime Enterprise.
- Licensing
- Global Settings
- Planning the Deployment
  A high-level overview of the items to consider when planning an HPE Ezmeral Runtime Enterprise deployment.
- System Requirements
- Deploying the Platform
  The topics in this section describe deploying HPE Ezmeral Runtime Enterprise. Deployment is divided into phases.
- Upgrading to HPE Ezmeral Runtime Enterprise 5.6.x
  This article describes the process to upgrade to the latest 5.6.x version of HPE Ezmeral Runtime Enterprise.
- Upgrading from HPE Ezmeral Runtime Enterprise Essentials
  Upgrade from HPE Ezmeral Runtime Enterprise Essentials to the full-featured HPE Ezmeral Runtime Enterprise or to HPE Ezmeral ML Ops by uploading a license. No additional steps are required.
- Manually Restarting HPE Ezmeral Runtime Enterprise Services
  This topic describes restarting HPE Ezmeral Runtime Enterprise services in non-Kubernetes hosts.
- Uninstalling and Reinstalling HPE Ezmeral Runtime Enterprise
- Support and Troubleshooting
App Workbench 5.1

Spark History Server

This topic provides an overview of Spark History Server.

Spark History Server provides a web UI to monitor and view the status of submitted Spark Applications. It shows the status of Running, Completed, and Failed (completed but failed) Spark Applications.

Spark History Server pulls the details of Spark Applications from event logs directory when you enable the event logging to submit the Spark Applications. The event logs are written to the event log directory on persistent storage.

You can access the Spark History Server web UI through node IP address and node port number as exposed by NodePort Service at

http://<node-IP-address>:<node-port-number>

The default node port number is 18080.

For more details about Spark History Server, see Apache Monitoring and Instrumentation.

Figure 1. Spark History Server Architecture