Installing Airflow

Describes how to install Airflow on a Kubernetes cluster in HPE Ezmeral Runtime Enterprise.

Prerequisites

  • For system, computation, and storage requirements, see Airflow Requirements.

  • Authentication of the Kubernetes cluster must be set to AD or LDAP.

  • You must have SSH access to the Kubernetes master node.

  • Required access rights: Kubernetes Cluster Administrator

Procedure

  1. Enable Airflow installation on the Kubernetes cluster by doing one of the following:

    • If the Kubernetes cluster has not been created, during Kubernetes cluster creation, deploy Airflow by selecting Enable Airflow in the Application Configurations tab.

      (Optional) To run Spark workflows, select Enable Spark Operator.

      For information about creating a Kubernetes cluster, see Creating a New Kubernetes Cluster.

    • If the Kubernetes cluster exists, deploy Airflow as follows:
      1. Edit the Kubernetes cluster, as described in Editing an Existing Kubernetes Cluster.
      2. On the Application Configurations tab, select Enable Airflow.

        (Optional) To run Spark workflows, select Enable Spark Operator.

      3. Save your changes.
      4. Wait until the Kubernetes cluster is ready and Airflow services are up and running.
  2. Create the Airflow cluster. Select one of the following methods:
    • Creating an Airflow Cluster Automatically: Use this method to create an Airflow cluster through the HPE Ezmeral Runtime Enterprise UI. This is the recommended Airflow cluster creation method.

      To create the Airflow cluster automatically, see Creating an Airflow Cluster Automatically.

    • Creating an Airflow Cluster Manually: Use this method to perform extra tuning of your Airflow cluster. For example, if you are using a proxy server that requires authentication.

      To create the Airflow cluster manually, see Creating an Airflow Cluster Manually.