Securely Passing Spark Configuration Values

Describes how to pass the sensitive data to Spark configuration using the Kubernetes Secret.

About this task

You can pass the sensitive data which are part of the Spark configuration using the Kubernetes secret. The secret has a Key-Value format where the key is spark-defaults.conf file and the value is sensitive data. You can use notebook to create secrets.

Procedure

  1. Create a Kubernetes Secret with the key as spark-defaults.conf and the value as sensitive data. See Creating a Secret.
  2. Add spark.mapr.extraconf.secret option with value as Secret name on Spark application YAML.

Example

  1. To securely pass the sensitive data, create a file with Spark configuration properties :
    cat << EOF > spark-defaults.conf
spark.hadoop.fs.s3a.access.key EXAMPLE_ACCESS_KEY
spark.hadoop.fs.s3a.secret.key EXAMPLE_SECRET_KEY
EOF
  2. Create a Secret from the file:
    kubectl create secret generic <k8s-secret-name> --from-file=spark-defaults.conf
  3. Set the spark.mapr.extraconf.secret option with Secret name in Spark application YAML.
    ...
spec:
  sparkConf:
    spark.mapr.extraconf.secret: "<k8s-secret-name>"
...