wuuker/docker-airflow repository overview

⁠ATTENTION - For users prior to 2.2.0 upgrading to 2.2.0

• There exists a breaking change⁠ that MUST BE ADDRESSED BEFORE UPGRADE which requires human intervention, airflow db upgrade command cannot solve this change if encountered. There are several way to address the problem:
  • follow the steps in the official document (link above), or
  • pause all DAGs, delete all running tasks and task record (this MAY work but not tested), or
  • if you are fine losing all records, you can ask Airflow to create a new DB from scratch:
    1. export all your connections and other settings
    2. delete the metadata DB
    3. recreate the container with new version of Airflow
    4. wait and let Airflow creates a new DB during startup
    5. import the settings you just exported.
  • (I accidentally destroyed the whole metadata DB stored in MariaDB while dealing with this.)
• Several parameters in airflow.cfg have been renamed, check the document⁠ to ensure that the name of the parameter(s) you passed into the container has the same name and take actions accordingly.

⁠Note

• Since I've not tested the image for too long, you may send feedback on my forked GitHub Repo⁠

⁠Usage

By default, docker-airflow runs Airflow with SequentialExecutor :

docker run -d -p 8080:8080 wuuker/docker-airflow webserver

If you want to run another executor, use the other docker-compose.yml files provided in this repository.

For LocalExecutor :

docker-compose -f docker-compose-LocalExecutor.yml up -d

For CeleryExecutor :

docker-compose -f docker-compose-CeleryExecutor.yml up -d

NB : If you want to have DAGs example loaded (default=False), you've to set the following environment variable :

LOAD_EX=n

docker run -d -p 8080:8080 -e LOAD_EX=y wuuker/docker-airflow

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values (equivalent to values in airflow.cfg/docker-compose*.yml) :

• Host : postgres
• Schema : airflow
• Login : airflow
• Password : airflow

⁠Configuring Airflow

It's possible to set any configuration value for Airflow from environment variables, which are used over values from the airflow.cfg.

The general rule is the environment variable should be named AIRFLOW__<section>__<key>, for example AIRFLOW__CORE__SQL_ALCHEMY_CONN sets the sql_alchemy_conn config option in the [core] section.

Check out the Airflow documentation⁠ for more details

You can also define connections via environment variables by prefixing them with AIRFLOW_CONN_ - for example AIRFLOW_CONN_POSTGRES_MASTER=postgres://user:password@localhost:5432/master for a connection called "postgres_master". The value is parsed as a URI. This will work for hooks etc, but won't show up in the "Ad-hoc Query" section unless an (empty) connection is also created in the DB

⁠Custom Airflow plugins

Airflow allows for custom user-created plugins which are typically found in ${AIRFLOW_HOME}/plugins folder. Documentation on plugins can be found here⁠

In order to incorporate plugins into your docker container

• Create the plugins folders plugins/ with your custom plugins.
• Mount the folder as a volume by doing either of the following:
  • Include the folder as a volume in command-line -v $(pwd)/plugins/:/usr/local/airflow/plugins
  • Use docker-compose-LocalExecutor.yml or docker-compose-CeleryExecutor.yml which contains support for adding the plugins folder as a volume

⁠Install custom python package

• Create a file "requirements.txt" with the desired python modules
• Mount this file as a volume -v $(pwd)/requirements.txt:/requirements.txt (or add it as a volume in docker-compose file)
• The entrypoint.sh script execute the pip install command (with --user option)

⁠UI Links

• Airflow: localhost:8080⁠
• Flower: localhost:5555⁠

⁠Scale the number of workers

Easy scaling using docker-compose:

docker-compose -f docker-compose-CeleryExecutor.yml scale worker=5

This can be used to scale to a multi node setup using docker swarm.

⁠Running other airflow commands

If you want to run other airflow sub-commands, such as list_dags or clear you can do so like this:

docker run --rm -ti wuuker/docker-airflow airflow list_dags

or with your docker-compose set up like this:

docker-compose -f docker-compose-CeleryExecutor.yml run --rm webserver airflow list_dags

You can also use this to run a bash shell or any other command in the same environment that airflow would be run in:

docker run --rm -ti wuuker/docker-airflow bash
docker run --rm -ti wuuker/docker-airflow ipython

⁠Simplified SQL database configuration using PostgreSQL

If the executor type is set to anything else than SequentialExecutor you'll need an SQL database. Here is a list of PostgreSQL configuration variables and their default values. They're used to compute the AIRFLOW__CORE__SQL_ALCHEMY_CONN and AIRFLOW__CELERY__RESULT_BACKEND variables when needed for you if you don't provide them explicitly:

You can also use those variables to adapt your compose file to match an existing PostgreSQL instance managed elsewhere.

Please refer to the Airflow documentation to understand the use of extras parameters, for example in order to configure a connection that uses TLS encryption.

Here's an important thing to consider:

Therefore you must provide extras parameters URL-encoded, starting with a leading ?. For example:

POSTGRES_EXTRAS="?sslmode=verify-full&sslrootcert=%2Fetc%2Fssl%2Fcerts%2Fca-certificates.crt"

⁠Simplified Celery broker configuration using Redis

If the executor type is set to CeleryExecutor you'll need a Celery broker. Here is a list of Redis configuration variables and their default values. They're used to compute the AIRFLOW__CELERY__BROKER_URL variable for you if you don't provide it explicitly:

You can also use those variables to adapt your compose file to match an existing Redis instance managed elsewhere.

I'd bind some directories in the container, listed below, to somewhere on my host machine, so I can my run own DAGs, keep event logs, and add plugins:

• /usr/local/airflow/dags
• /usr/local/airflow/plugins
• /usr/local/airflow/logs