Airflow Requirements

Describes the system, computation, and storage requirements for Airflow clusters.

System Requirements

  • Pod DNS Domain is cluster.local.

  • Cluster authentication is configured as AD or LDAP.

Computation Requirements

You can use DAGs to create many workers that can run simultaneously. In general, for an Airflow cluster with workers, approximately 5 CPUs and 5000Mi memory are needed.

Specifically, the following resources must be available:

  • To install the Airflow Operator and Base during cluster creation or editing:

    • CPU: 1800m

    • Memory: 4100Mi

  • To install Airflow cluster per tenant:

    • CPU: 3000m

    • Memory: 4000Mi

  • To launch one worker:

    • CPU: 1500m

    • Memory: 712Mi

Storage Requirements

Airflow requires dynamic-volume provisioning. Persistent storage must be configured and registered in the HPE Ezmeral Runtime Enterprise deployment. For information, see Storage.