Managing Spark on Kubernetes Using Helm Charts and Spark Operator

Starting with the DEP 10.1.1 release, support for the Spark Operator has been added, enabling Kubernetes-native management of Spark workloads through SparkApplication custom resources. This topic provides detailed information about Data Fabric's Helm and Spark integration, including installation steps.

Helm charts are packages of pre-configured Kubernetes resources that simplify the deployment, versioning, and management of applications on Kubernetes clusters. They bundle all the necessary YAML configuration files, templates, and dependencies into a single, reusable package that can be installed with a single command using the Helm CLI.

Starting with DEP 10.1.1, the Spark Operator is introduced for Spark on Data Fabric, enabling Kubernetes-native management of Spark workloads through SparkApplication custom resources. The Spark Helm chart enables containerized deployment of Spark applications to Kubernetes clusters (such as OpenShift Container Platform) by packaging the Spark Operator, DFClient Operator, required RBAC configurations, secrets, and Spark runtime settings. This allows users to run Spark shells and submit Spark applications declaratively as SparkApplication custom resources using Kubernetes as the cluster manager, with Data Fabric providing the underlying data layer.

This topic outlines the steps to set up and deploy a Data Fabric Client on a Kubernetes cluster, including installing prerequisites, configuring the DFClient Operator, generating secrets, and running Spark applications.

For the latest Helm chart files and instructions, refer to the Commits · mapr/data-fabric-helm-charts branch of the data-fabric-helm-charts repository. See the Release release/fy26-q3 · mapr/data-fabric-helm-charts tag for the current release.
NOTE
As of DEP 10.1.1, Tenant Operator has been renamed to DFClient Operator.

Prerequisites

  • Kubernetes Cluster: A running Kubernetes cluster with the necessary permissions to deploy resources to the cluster.
  • kubectl: Ensure that you have kubectl installed and configured to interact with your Kubernetes cluster.
  • Helm: Ensure you have Helm version 3.x or later installed.
  • Container Registry Access: Ensure that you have access to the container registry containing the required images.
  • Cluster Admin Privileges: Ensure that you have privileges for creating cluster-wide resources.

You can find all necessary images in the https://hub.docker.com/u/maprtech repository.

1. Install and Configure the DFClient Operator and related components

Cert-Manager is required for managing certificates within the cluster.
  • Apply Cert-Manager CRDS:
    kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml
  • Add the Jetstack Helm Repository and Update:
    helm repo add jetstack https://charts.jetstack.io
helm repo update
  • Install Cert-Manager in the 'cert-manager' Namespace:
    helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.8.0

2. Add Registry Certificate on Nodes

Install the registry certificate on all cluster nodes according to your internal security procedures.
NOTE
The exact steps for this will vary depending on your specific environment. Refer to your internal documentation for guidance.

3. Install DFClient Operator

Install the DFClient Operator using Helm:
helm install dfclient-operator dfclient-operator-chart/ \
    -n hpe-dfclient \
    --create-namespace \
    -f dfclient-operator-chart/values.yaml

4. Generate External Secrets

Run the external secret generation script (gen-external-secrets.sh) on a cluster node. This script is assumed to be located in your environment.

5. Apply External Secrets and Additonal Configurations

a. Apply External Secrets
  • Apply the generated external secrets:
    kubectl apply -f /tmp/mapr-external-secrets.yaml
b. (Optional) Create an ImagePull Secret
  • Replace your-base64-encoded-docker-config with your actual encoded docker configuration:
    cat <<EOF > imagepull-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: imagepull
  namespace: sampledfclient
  labels:
    hpe.com/cluster: none
    hpe.com/component: imagepull
    hpe.com/namespacetype: DFClient
    hpe.com/dfclient: sampledfclient
data:
  .dockerconfigjson: "<your-base64-encoded-docker-config>"
type: kubernetes.io/dockerconfigjson
EOF

# Apply the secret
kubectl apply -f imagepull-secret.yaml
  • Log in to your Docker registry, generate the base-64-encoded docker configuration, and encode the Docker configuration file:
    docker login <your-registry>
cat ~/.docker/config.json | base64 -w 0

6. Deploy the DFClient

Deploy a DFClient using the provided example configuration: 
kubectl apply -f dfclient-examples/external-full.yaml
NOTE
Customize the DFClient configuration according to your requirments before applying it.

Running Spark Applications

You can run Spark applications using either the Spark Operator or the Data Fabric Client CLI.

Running with Spark Operator

a. Install the Spark Operator
helm -n hpe-spark-operator install hpe-spark-operator --create-namespace ./spark-operator-chart/
NOTE
More details on Spark Operator are available in the README file for spark-operator-chart.
b. Run the Ticket Creator Script
  • Navigate to the dfclient CLI pod and execute the bundled script:
    ticketcreator.sh
c. Fill in the Spark Application Custom Resource with your values
  • A sample Custom Resource for a Spark Application is located in the spark-examples directory.

    Set the spark.mapr.user.secret field to the name of the secret generated by the ticketcreator.sh script in the previous step.

Running with the Data Fabric Client CLI

Option 1: Using Ticket Authentication

a. Run the Ticket Creator Script
  • Navigate to the dfclient CLI pod and execute the bundled script:
    ticketcreator.sh

This script generates a Kubernetes secret containing the authentication ticket.

b. Launch the Spark Shell with Ticket Auhtentication
/opt/mapr/spark/spark-3.5.5/bin/spark-shell \
  --master k8s://https://<kubernetes-controller-host>:6443 \
  --conf spark.executor.instances=2 \
  --conf spark.mapr.user.secret=<generated-secret-name> \
  --conf spark.kubernetes.container.image=<spark-image> \
  --conf spark.kubernetes.namespace=<dfclient-namespace> \
  --conf spark.kubernetes.container.image.pullPolicy=Always \
  --conf spark.mapr.cluster.configMap=cluster-cm \
  --conf spark.authenticate=false \
  --conf spark.authenticate.enableSaslEncryption=false \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=hpe-<dfclient-namespace>
Replace the following placeholders:
  • <kubernetes-controller-host> - The hostname or IP address of your Kubernetes controller
  • <generated-secret-name> - The name of the secret generated by the ticketcreator.sh script
  • <spark-image> - The full name of the Spark container image (for example, docker.io/maprtech/df-apache-spark:8.1.0_10.1.1_edf
  • <dfclient-namespace - The namespace where your dfclientis deployed (for example, sampledfclient

Option 2: Using JWT Token Authentication

a. Generate a JWT Token

Use the following curl command to generate your JWT token:
curl -k -d 'client_id=edf-client' \
  -d 'client_secret=<your-client-secret>' \
  -d 'username=<your-username>' \
  -d 'password=<your-password>' \
  -d 'grant_type=password' \
  'https://<sso-node>:6443/realms/master/protocol/openid-connect/token' | \
  jq

Replace the placeholder values with your actual authentication information.

b. Create a Secret with JWT Tokens
# Extract tokens from the JWT response
ACCESS_TOKEN=$(echo $JWT_RESPONSE | jq -r '.access_token')
REFRESH_TOKEN=$(echo $JWT_RESPONSE | jq -r '.refresh_token')

# Create a secret with the tokens
cat <<EOF > jwt-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: <token-secret-name>
  namespace: <dfclient-namespace>
type: Opaque
data:
  CLUSTER_NAME: $(echo -n "<your-cluster-name>" | base64 -w 0)
  JWT_REFRESH_TOKEN: $(echo -n "$REFRESH_TOKEN" | base64 -w 0)
  JWT_TOKEN: $(echo -n "$ACCESS_TOKEN" | base64 -w 0)
EOF

# Apply the secret
kubectl apply -f jwt-secret.yaml
Launch the Spark Shell with JWT Authentication
/opt/mapr/spark/spark-3.5.5/bin/spark-shell \
  --master k8s://https://<kubernetes-controller-host>:6443 \
  --conf spark.executor.instances=2 \
  --conf spark.mapr.jwt.secret=<token-secret-name> \
  --conf spark.kubernetes.container.image=<spark-image> \
  --conf spark.kubernetes.namespace=<dfclient-namespace> \
  --conf spark.kubernetes.container.image.pullPolicy=Always \
  --conf spark.mapr.cluster.configMap=cluster-cm \
  --conf spark.authenticate=false \
  --conf spark.authenticate.enableSaslEncryption=false \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=<dfclient-namespace>
Replace the following placeholders:
  • <kubernetes-controller-host> - The hostname or IP address of your Kubernetes controller
  • <token-secret-name> - The name of the secret containing your JWT tokens
  • <spark-image> - The full name of the Spark container image
  • <dfclient-namespace> - The namespace where your dfclient is deployed

For the github readme instructions for integrating Helm Charts for Spark, see DFClient Operator Installation Guide.