Installing and Configuring Spark History Server

This section describes how to install and configure Spark History Server on HPE Ezmeral Runtime Enterprise.

Prerequisites

  1. Log in as a Kubernetes Tenant Administrator or a Kubernetes Tenant Member in HPE Ezmeral Runtime Enterprise.
  2. Install Spark Operator and enable the webhook. See Spark Operator.
  3. If you are using the PVC:
    1. Configure the PVC of type ReadWriteMany in the tenant namespace.
    2. To ensure you can write data to the file system storage (for example, maprfs) of PV, set the permissions to 777 on the target folder. Platform Administrator can set the permissions on the target folder of the file system storage.

About this task

In HPE Ezmeral Runtime Enterprise, you can install Spark History Server using GUI or manually using the Helm chart for Apache Spark.

Apache Spark supports the Data Fabric filesystem (maprfs) and PersistentVolumeClaim (PVC) as the persistent storage on HPE Ezmeral Runtime Enterprise.

Learn more about supported Spark versions at Interoperability Matrix for Spark.

Installing Spark History Server Using the GUI

About this task

Install the Spark History Server by using the HPE Ezmeral Runtime Enterprise GUI.

Procedure

  1. Log in as a Kubernetes Tenant Administrator or a Kubernetes Tenant Member on the HPE Ezmeral Runtime Enterprise GUI.
  2. Click Applications in the main menu. You will see Kubernetes Applications tiles under KubeDirector tab.
  3. Navigate to Spark History Server tile and click Launch.
  4. Configure Cluster Detail and Settings on Create Application screen.
    Cluster Detail:
    Enter the Name and Description of the application.
    Settings:

    Set the CPU and Memory (GB) resources.

    To set Event Log Storage Settings, check Event Log Storage and select Storage Type.

    To set the additional configuration options not available through HPE Ezmeral Runtime Enterprise GUI , edit the values.yaml file and install Spark History Server using the helm charts as described in Installing Spark History Server Using the Helm section.

  5. To view yaml, click Edit/Launch yaml.
  6. Click Submit.

Results

The GUI installs the Spark History Server in a tenant namespace.

Installing Spark History Server Using the Helm

Prerequisites

  1. Install and configure Helm 3.
  2. Install and configure kubectl.

About this task

Install the Spark History Server on Data Fabric tenants which are HPE Ezmeral Data Fabric on Kubernetes tenants or HPE Ezmeral Data Fabric on Bare Metal tenants or non Data Fabric tenants by using the Helm chart.

Procedure

Helm install the Spark History Server on HPE Ezmeral Runtime Enterprise:
  • Installing Spark History Server on Data Fabric tenants:
    To helm install the Spark History Server, run the following command:
    helm dependency update ./<path-to-spark-hs-chart>
    helm install <spark-hs-name> ./<path-to-spark-hs-chart>

    The helm install creates the helm chart in the default namespace. To create the chart in a different existing namespace, use the flag -n <tenant-namespace>.

    To set the tenant namespace during installation, use the flag --set tenantNameSpace=<tenant-namespace>.

    To configure PVC, set the following flags:
    --set pvc.enablePVC=true --set pvc.ExistingClaimName=<pvc-name> --set pvc.eventsDir=<path-to-directory>

    Alternatively, you can configure the PVC in the values.yaml file.

  • Installing Spark History Server on non Data Fabric tenants:
    To helm install the Spark History Server for the tenant type none, run the following command. You must use PVC as a persistent storage for event logs for the tenant type none. Create a PVC to start a Spark History Server pod and set the PVC name for the event log storage. 
    helm install -f <path-to-values.yaml-file> <spark-hs-name> <path-to-spark-history-server-chart> \
--namespace <tenant-namespace> \
--set tenantNameSpace=<tenant-namespace>  \
--set tenantIsUnsecure=true \
--set eventlogstorage.kind=pvc \
--set eventlogstorage.pvcname=<pvc-name>

    See Using Custom KeyStore.

Running the helm install installs Spark History Server in a tenant namespace.