Public | Automated Build

Last pushed: 10 months ago
Short Description
docker-airflow
Full Description

docker-airflow for python3




This repository contains Dockerfile of airflow for Docker's automated build published to the public Docker Hub Registry.

Informations

Installation

Pull the image from the Docker repository.

    docker pull harmy/docker-airflow

Build

For example, if you need to install Extra Packages, edit the Dockerfile and than build-it.

    docker build --rm -t harmy/docker-airflow .

Usage

By default, docker-airflow run Airflow with SequentialExecutor :

    docker run -d -p 8080:8080 harmy/docker-airflow

If you want to run other executor, you've to use the docker-compose.yml files provided in this repository.

For LocalExecutor :

    docker-compose -f docker-compose-LocalExecutor.yml up -d

For CeleryExecutor :

    docker-compose -f docker-compose-CeleryExecutor.yml up -d

NB : If you don't want to have DAGs example loaded (default=True), you've to set the following environment variable :

LOAD_EX=n

    docker run -d -p 8080:8080 -e LOAD_EX=n harmy/docker-airflow

If you want to use Ad hoc query, make sure you've configured connections:
Go to Admin -> Connections and Edit "mysql_default" set this values (equivalent to values in airflow.cfg/docker-compose.yml) :

  • Host : mysql
  • Schema : airflow
  • Login : airflow
  • Password : airflow

For encrypted connection passwords (in Local or Celery Executor), you must have the same fernet_key. By default docker-airflow generates the fernet_key at startup, you have to set an environment variable in the docker-compose (ie: docker-compose-LocalExecutor.yml) file to set the same key accross containers. To generate a fernet_key :

    python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print FERNET_KEY"

Check Airflow Documentation

Install custom python package

  • Create a file "requirements.txt" with the dedired python modules
  • Mount this file as a volume -v $(pwd)/requirements.txt:/requirements.txt
  • The entrypoint.sh script execute the pip install command (with --user option)

UI Links

When using OSX with boot2docker, use: open http://$(boot2docker ip):8080

Scale the number of workers

Easy scaling using docker-compose:

    docker-compose scale worker=5

This can be used to scale to a multi node setup using docker swarm.

Links

Wanna help?

Fork, improve and PR. ;-)

Docker Pull Command
Owner
harmy
Source Repository

Comments (0)