Configure the Airflow Helm Chart

Details how to customize values.yaml to deploy Apache Airflow on Kubernetes with a Data Fabric DEP environment.

This section describes how to configure values.yaml to deploy Apache Airflow on Kubernetes in an HPE Data Fabric environment, covering storage, database, executor, security keys, DAG distribution, and Data Fabric authentication integration.

Set Persistent Volume storage classes

Set Persistent Volume (PV) storage classes for:
  • workers - workers.persistence.storageClassName or workers.celery.persistence.storageClassName
  • triggerer - triggerer.persistence.storageClassName
  • redis - redis.persistence.storageClassName (CeleryExecutor only)

Configure the Airflow metadata database

For development / testing purposes, the PostgreSQL Helm Chart, which is automatically deployed, can be used. PostgreSQL Helm chart is included in the Airflow Helm chart. It requires primary.persistence.storageClass for Persistent Volume.

Default configuration:
# Configuration for postgresql subchart
# Uses bitnamilegacy images to avoid Bitnami licensing restrictions
# Not recommended for production - use external database instead
postgresql:
  enabled: true
  image:
    repository: bitnamilegacy/postgresql
    tag: "16.1.0-debian-11-r15"
  auth:
    enablePostgresUser: true
    postgresPassword: postgres
    username: ""
    password: ""
For production, the extrnal DB should be configured in Airflow values.yaml. Example:
  metadataConnection:
    user: postgres
    pass: postgres
    protocol: postgresql
    host: ~
    port: 5432
    db: postgres

Set the Airflow executor

The Airflow executor can be one or more of the following: LocalExecutor, CeleryExecutor, KubernetesExecutor. Example:
executor: "CeleryExecutor"
  • LocalExecutor - Tasks execute as sub-processes within the Scheduler pod (suitable for small workloads / development)..
  • CeleryExecutor - Tasks are distributed across persistent worker pods via a Redis message broker (redis.persistence.storageClassName required. See https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html for more information.
  • KubernetesExecutor - Launches a dedicated, ephemeral worker pod for each task instance directly via the Kubernetes API.

Configure Production Security Keys

For production environments, set the fernetKey. The Fernet key is used to encrypt passwords. It can only be set during install, not upgrade.
IMPORTANT
The fernetKey must be generated and configured during the intitial helm install operation. Changing or rotating fernetKey during a subsequent helm upgrade operation without manually re-encrypting database connections will render existing encrypted credentials unreadable.

For production environments, also set the apiSecretKey. The Flask secret key is used to enable the Airflow API to encrypt the browser session.

Configure DAG Storage and Synchronization

Configure the DAGs volume to synchronize DAGs to Airflow pods. Airflow Helm chart has a few options. See the dags section in values.yaml or the Apache Airflow Helm chart documentation.

Configure MapR-SASL Authentication ticket (User Ticket Secret)

By default, Airflow uses a maprserver ticket to communicate with the external cluster. But it is possible to use a user ticket for MapR-SASL authentication in tasks.

To do this, you must create a secret in the Airflow namespace with user ticket and configure it:
# HPE DF security for non-admin user. This is a secret for user ticket that allows running Airflow services
# with non-admin user using MapR-SASL.
dep:
  # Configuration for user MapR-SASL authentication
  userTicket:
    # Use user ticket for MapR-SASL, instead of default
    useMaprTicket: true
    # Secret name where user ticket is stored
    ticketSecret: <secret_name>
    # Data key which uses for ticket content in the secret
    ticketValue: <ticket_key>
Example:
# cat /tmp/maprticket_1000 | base64 -w 0
pjTu21aV6jtPNPxL3C7rJDvBAkBI94qI

# vi ticket.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ticket-secret
type: Opaque
data:
  ticket: pjTu21aV6jtPNPxL3C7rJDvBAkBI94qI

# kubectl -n <namespace> apply -f ticket.yaml
............................................................
dep:
  # Configuration for user MapR-SASL authentication
  userTicket:
    # Use user ticket for MapR-SASL, instead of default
    useMaprTicket: true
    # Secret name where user ticket is stored
    ticketSecret: ticket-secret
    # Data key which uses for ticket content in the secret
    ticketValue: ticket

Configure SSO authentication for Airflow services

In order for Airflow to use external OpenID Connect (OIDC) for authentication, you must configure the following options in values.yaml:
# HPE DF security for non-admin user. This is a secret for user ticket that allows running Airflow services
# with non-admin user using MapR-SASL.
dep:
.................................................
  clusterConf:
.................................................
    # Airflow JWT configuration for webserver UI.
    externalJWT: true
    # -- The authentication server URL. Must be provided if it uses external JWT for UI authentication.
    authServerUrl: ~ # https://auth.example.com/realms/master
    # -- The client to use when authenticating with the authentication server.
    client:
      # -- The client ID to use when contacting the authentication server's endpoint to validate token.
      id: ~ # example: edf-client
      # -- The secret to pull the client secret from.
      secret:
        # -- The name of the secret to pull the client secret from.
        name: oidc-secret
        # -- The key name inside the secret to pull the client secret from.
        key: oidcsecretkey