Submitting Spark Applications Using spark-submit

This topic describes how to install spark-client Helm chart and submit Spark applications using spark-submit utility in HPE Ezmeral Runtime Enterprise.

In HPE Ezmeral Runtime Enterprise, you can install spark-client manually using the Helm chart for Apache Spark.

Learn more about supported Spark versions at Interoperability Matrix for Spark.

Install spark-client Helm Chart

To install the Helm chart for spark-client for Apache Spark 3.x.x in an existing tenant namespace, run:
helm install <spark-client-name> ./<path-to-spark-client-chart> -n <tenant-namespace>
To install the Helm chart for spark-client for Apache Spark 3.x.x in a new tenant namespace, run:
helm install <spark-client-name> ./<path-to-spark-client-chart> -n <tenant-namespace> --create-namespace
To install the Helm chart for spark-client for Apache Spark 2.x.x in an existing namespace, run 
helm install <spark-client-name> ./<path-to-spark-client-chart> -n <tenant-namespace> /
--set image.imageName=<spark-client-image-name> /
--set image.tag=<spark-client-imagetag>
See Spark Images.

Submit Spark Applications Using spark-submit on Cluster Mode

You can configure and submit the Spark applications using the spark-submit on cluster deploy mode in HPE Ezmeral Runtime Enterprise.

When your spark-client pod is up and running, perform the following steps:
  1. Determine the gateway FQDN and port to SSH into the client pod. Run:
    kubectl describe svc spark-client-spark-client-chart -n t1 | grep gateway\
  2. SSH into that FQDN and port, and authenticate as a tenant member AD user.
  3. When you are in the spark-client pod, run maprlogin password and enter the AD password again.
  4. Locate the Spark binary of your choice under /opt/mapr/spark/spark-<version>/bin. See Interoperability Matrix for Spark.
  5. Submit the Spark applications. See Launching Applications with spark-submit.
NOTE
HPE Ezmeral Runtime Enterprise does not support client deploy mode.

Delete spark-client Helm Chart

Run the following command to delete the spark-client Helm chart.
helm delete <spark-client-name> -n <tenant-namespace>
For example:
helm delete spark-client -n sampletenant