Helm Chart Integration for Airflow

Documents the Airflow Helm Chart introduced in DEP 10.2.0, covering Kubernetes-native deployment of Apache Airflow on Data Fabric.

Starting with the DEP 10.2.0 release, support for Airflow Helm Chart has been added, enabling Kubernetes-native deployment and management of Apache Airflow workflow orchestration. This topic provides detailed information about Data Fabric's Helm and Airflow integration, including prerequisite setup, security and executor configuration, and installation steps.

Helm chartsare packages of pre-configured Kubernetes resources that simplify the deployment, versioning, and management of applications on Kubernetes clusters. They bundle all the necessary YAML configuration files, templates, and dependencies into a single, reusable package that can be installed with a single command using the Helm CLI.

Starting with DEP 10.2.0, the Airflow Helm Chart is introduced for Data Fabric, enabling containerized deployment of Airflow components (Schedulers, Webservers, Workers, and Triggerers) to Kubernetes clusters. The Airflow Helm chart integrates with the Data Fabric Client (DFClient) Operator and supports MapR-SASL ticket-based authentication, SSO/JWT web UI authentication, persistent storage configurations, and flexible Airflow task execution (LocalExecutor, CeleryExecutor, and KubernetesExecutor), with Data Fabric providing the underlying data platform and storage layer.

This topic outlines the steps to set up, configure, and deploy the Data Fabric Airflow Helm Chart on a Kubernetes cluster, including configuring storage classes, metadata databases, authentication settings, and running the deployment commands.

For the latest Helm chart files and instructions, refer to the Commits · mapr/data-fabric-helm-charts branch of the data-fabric-helm-charts repository. See the Release release/fy26-q3 · mapr/data-fabric-helm-charts tag for the current release.
NOTE
As of DEP 10.1.1, Tenant Operator has been renamed to DFClient Operator.

Prerequisites

  • Deploy DFClient Operator on a Kubenetes cluster.
  • Generate external secrets.
  • Apply external secrets and additional configurations.
  • Deploy a DFClient

Examples for DFClient configuration and deployment

Configuration:
apiVersion: hcp.hpe.com/v1
kind: DFClient
metadata:
  name: sampledfclient
spec:
  clustername: hpe.cluster.com
  clustertype: external
  baseimagetag: ""
  imageregistry: docker.io/maprtech
  imagepullsecret: imagepull
  loglocation: /var/log/mapr/
  corelocation: /var/log/mapr/cores
  podinfolocation: /var/log/mapr/podinfo
  security:
    environmenttype: hcp
    dnsdomain: cluster.local
    usesssd: true
    externalconfigmap: mapr-external-cm
    externalusersecret: mapr-user-secrets
    externalserversecret: mapr-server-secrets
    externalclientsecret: mapr-client-secrets
    systemusersecret: system-user-secrets
    sslsecret: ssl-secrets
    podsecuritypolicy: dfclient-psp
    role: dfclient-role
    userrole: dfclient-userrole
    terminalrole: dfclientcli-role
  debugging:
    loglevel: INFO
    preservefailedpods: true
    wipelogs: false
  userlist:
    - mapr
  grouplist:
    - mapr
Deploy command:
kubectl apply -f <tenant_conf>.yaml

The Tenant Operator automatically creates a namespace with the same name as the DFClient and applies all secrets and ConfigMap from the hpe-externalclusterinfo namespace.