Configuring a Remote MySQL Database for Airflow

This topic describes how to configure a remote MySQL database for Airflow on the HPE Ezmeral Data Fabric.

Prerequisites

To connect to MySQL from Airflow, install the mysqlclient by using these steps:
  1. Run . <airflow_home>/build/env/bin/activate
  2. Run pip install mysqlclient==2.2.0
  3. Run deactivate

About this task

Airflow uses SQLAlchemy to connect to the metadata database. The metadata database stores the information about Airflow configurations, user information, roles and policies, and statistics of each DAG state, run, and task.

Airflow supports MySQL database engine versions 5.7 and 8. For a list of the supported databases, see Choosing database backend.

Procedure

  1. To configure the remote MySQL Database for Airflow, see Setting Up a MYSQL Database.
  2. After you have configured a database user that Airflow can use to access the database and <AIRFLOW-HOME>/conf/airflow.cfg is updated with your SQLAlchemy connection string, run this command: 
    sql_alchemy_conn = mysql+mysqldb://<airflow-user>:<airflow-password>@<host>[:<port>]/<airflow-dbname>
  3. Create the database schema using the steps that apply to the currently-installed EEP. To identify the EEP that is installed, see Checking the EEP Version:
    • EEP 9.2.0 and later:
      1. Run the airflow db migrate command:
        airflow db migrate
      2. Run the airflow connections create-default-connections command:
        airflow connections create-default-connections
    • EEP 9.1.x and earlier:
      airflow db init
  4. After MySQL configuration is completed, create a user by following the steps in the Command Line Interface and Environment Variables Reference. For example: 
    airflow users create --username mapr --firstname mapr --lastname mapr -p mapr --role Admin --email admin@example.org
  5. Restart Airflow services as described in Starting, Stopping, and Restarting Airflow Services.