Configuring Resource Limits on Spark History Server

This section guides you through configuring resource limits for Spark History Server on ResourceQuota configured namespace.

A resource quota is defined by the ResourceQuota object. It limits the total consumption of compute resources(CPU, memory, etc.) per namespace. See Resource Quotas.

If you are using Spark History Server to monitor the Spark Applications on ResourceQuota configured namespace, you can configure resource limits for Spark History Server in values.yaml file. See values.yaml file in the Helm chart.

Configuring Resource Limits on Spark History Server

To configure resource limits for Spark History Server, set the resources property in values.yaml file.

For example:
resources:
   limits:
     cpu: 8000m
     memory: 8Gi
     ephemeral-storage: 30Gi
   requests:
     cpu: 2000m
     memory: 8Gi
     ephemeral-storage: 30Gi
You can update the resource limits and requests for Spark History Server after installation in two different ways:
  1. Update the resources property in values.yaml file.
  2. Run the helm upgrade command using -set flag.
    helm upgrade <spark-hs-name> ./<path-to spark-hs-chart> /  -n <your-namespace> --set resources.limits.cpu=<value>
    For example:
    helm upgrade spark-hs ./charts/spark-hs-2.4.7/spark-hs-chart/  -n sampletenant --set resources.limits.cpu=10000m
    The helm upgrade command will terminate the existing pod and create a new Spark History Server pod. Run the export command to get the new node IP and node port to access the Spark web UI.
    export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "spark-hs-chart.fullname" . }})
    export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") 
    echo http://$NODE_IP:$NODE_PORT