Public Repository

Last pushed: 2 months ago
Short Description
A docker image and kubernetes config files to run Airflow on Kubernetes
Full Description

kube-airflow



This repository contains:

  • Dockerfile(.template) of airflow for Docker images published to the public Docker Hub Registry.
  • airflow.all.yaml for creating Kubernetes services and deployments to run Airflow on Kubernetes

Informations

Installation

Create all the deployments and services for Airflow:

    kubectl create -f airflow.all.yaml

Build

git clone this repository and then just run:

    make build

Usage

Create all the deployments and services to run Airflow on Kubernetse:

   kubectl create -f airflow.all.yaml

It will create deployments for:

  • postgres
  • rabbitmq
  • airflow-webserver
  • airflow-scheduler
  • airflow-flower
  • airflow-worker

and services for:

  • postgres
  • rabbitmq
  • airflow-webserver
  • airflow-flower

You can browse the Airflow dashboard via running:

make browse-web

the Flower dashboard via running:

make browse-flower

If you want to use Ad hoc query, make sure you've configured connections:
Go to Admin -> Connections and Edit "mysql_default" set this values (equivalent to values in config/airflow.cfg) :

  • Host : mysql
  • Schema : airflow
  • Login : airflow
  • Password : airflow

Check Airflow Documentation

Run the test "tutorial"

    kubectl exec web-<id> --namespace airflow-dev airflow backfill tutorial -s 2015-05-01 -e 2015-06-01

Scale the number of workers

For now, update the value for the replicas field of the deployment you want to scale and then:

    make apply

Wanna help?

Fork, improve and PR. ;-)

Docker Pull Command
Owner
mumoshu

Comments (0)