Spark Operator

This topic provides an overview of Spark Operator on HPE Ezmeral Container Platform.

HPE Ezmeral Container Platform 5.3 provides separate Spark Operator for Apache Spark 2.4.7 and Apache Spark 3.1.1. You can submit Spark Applications for Apache Spark 2.4.7 and Apache Spark 3.1.1 only using the Spark Operator of the same version. Spark operator controls the workflow of Spark Applications by creating a driver pod. A Spark driver pod launches a set of Spark executors that execute the job you want to run.

HPE Ezmeral Container Platform supports all the features and parameters supported by open-source Spark on K8s documentation excluding the security feature. HPE Ezmeral Container Platform supports the following Spark security features:
  • If you are a local user, set the spark.mapr.user.secret option on your Spark application yaml file.
  • If you are AD/LDAP user, spark.mapr.user.secret option is automatically set using the ticketgenerator webhook.
  • You must not change the user context. See using pod security context.