Airflow version: Notation for representing version XX.YY.ZZ
Execution Mode: standalone(simple container for exploration purpose, based on sqlite as airflow metadata db & SequentialExecutor ) or prod(single node based, LocalExecutor amd mysql as airflow metadata db) and cluster (for distributed production long run use-cases, container runs as either server or worker )
Backend database: standalone- Sqlite, prod & cluster- Mysql
Scheduler: standalone- Sequential, prod- LocalExecutor and Cluster- Celery
Task queue: cluster- Redis
Log location: local file system (Default) or AWS S3 (through entrypoint-s3.sh)
User authentication: Password based & support for multiple users with superuser privilege.
Docker base image: debian
Code enhancement: password based multiple users supporting super-user(can see all dags of all owner) feature. Currently, Airflow is working on the password based multi user feature.
Other features: support for google cloud platform packages in container.
There are two docker files in the folder docker-files.
Base image(DockerFile-BaseXX.YY.ZZ) - file for building base image which consist of packages of airflow, java, redis and other basic components.
Working image(DockerFile-XX.YY.ZZ) - Depend on the base image. Build image with patches of airflow, creating user, installing gcp packages and setting up the working environment.
Airflow scheduler needs a restart after sometime for properly scheduling of the task. Shell script for restarting scheduler is present in folder config
Airflow container by default is configured for writing logs on local filesystem but can be configured for writing on AWS S3. AWS credentials needs to be updated in credentials file in folder config.