bitnami/airflow-worker
🛑 DEPRECATED Bitnami container image for Apache Airflow Worker
5M+
The Apache Airflow Worker container is no longer maintained upstream and is now deprecated. This image will no longer be released in our catalog, but already released container images will persist in the registries.
After some months, this repository will be removed in favor of bitnami/airflow-worker-archived where all existing images can be found.
Apache Airflow is a tool to express and execute workflows as directed acyclic graphs (DAGs). Airflow workers listen to, and process, queues containing workflow tasks.
Overview of Apache Airflow Worker Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.
docker run --name airflow-worker bitnami/airflow-worker:latest
You can find the default credentials and available configuration options in the Environment Variables section.
Looking to use Apache Airflow Worker in production? Try VMware Tanzu Application Catalog, the commercial edition of the Bitnami catalog.
Dockerfile
linksLearn more about the Bitnami tagging policy and the difference between rolling tags and immutable tags in our documentation page.
You can see the equivalence between the different tags by taking a look at the tags-info.yaml
file present in the branch folder, i.e bitnami/ASSET/BRANCH/DISTRO/tags-info.yaml
.
Subscribe to project updates by watching the bitnami/containers GitHub repo.
To run this application you need Docker Engine >= 1.10.0
. Docker Compose is recommended with a version 1.6.0
or later.
Airflow Worker is a component of an Airflow solution configuring with the CeleryExecutor
. Hence, you will need to rest of Airflow components for this image to work.
You will need an Airflow Webserver, an Airflow Scheduler, a PostgreSQL database and a Redis(R) server.
Create a network
docker network create airflow-tier
Create a volume for PostgreSQL persistence and create a PostgreSQL container
docker volume create --name postgresql_data
docker run -d --name postgresql \
-e POSTGRESQL_USERNAME=bn_airflow \
-e POSTGRESQL_PASSWORD=bitnami1 \
-e POSTGRESQL_DATABASE=bitnami_airflow \
--net airflow-tier \
--volume postgresql_data:/bitnami/postgresql \
bitnami/postgresql:latest
Create a volume for Redis(R) persistence and create a Redis(R) container
docker volume create --name redis_data
docker run -d --name redis \
-e ALLOW_EMPTY_PASSWORD=yes \
--net airflow-tier \
--volume redis_data:/bitnami \
bitnami/redis:latest
Launch the Apache Airflow Worker web container
docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e AIRFLOW_EMAIL=user@example.com \
--net airflow-tier \
bitnami/airflow:latest
Launch the Apache Airflow Worker scheduler container
docker run -d --name airflow-scheduler \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
--net airflow-tier \
bitnami/airflow-scheduler:latest
Launch the Apache Airflow Worker worker container
docker run -d --name airflow-worker \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_QUEUE=new_queue \
--net airflow-tier \
bitnami/airflow-worker:latest
Access your application at http://your-ip:8080
docker-compose.yaml
curl -LO https://raw.githubusercontent.com/bitnami/containers/main/bitnami/airflow/docker-compose.yml
docker-compose up
Please be aware this file has not undergone internal testing. Consequently, we advise its use exclusively for development or testing purposes. For production-ready deployments, we highly recommend utilizing its associated Bitnami Helm chart.
If you detect any issue in the docker-compose.yaml
file, feel free to report it or contribute with a fix by following our Contributing Guidelines.
The Bitnami Airflow container relies on the PostgreSQL database & Redis to persist the data. This means that Airflow does not persist anything. To avoid loss of data, you should mount volumes for persistence of PostgreSQL data and Redis(R) data
The above examples define docker volumes namely postgresql_data
, and redis_data
. The Airflow application state will persist as long as these volumes are not removed.
To avoid inadvertent removal of these volumes you can mount host directories as data volumes. Alternatively you can make use of volume plugins to host the volume data.
Mount host directories as data volumes with Docker Compose
The following docker-compose.yml
template demonstrates the use of host directories as data volumes.
version: '2'
services:
postgresql:
image: 'bitnami/postgresql:latest'
environment:
- POSTGRESQL_DATABASE=bitnami_airflow
- POSTGRESQL_USERNAME=bn_airflow
- POSTGRESQL_PASSWORD=bitnami1
volumes:
- /path/to/postgresql-persistence:/bitnami
redis:
image: 'bitnami/redis:latest'
environment:
- ALLOW_EMPTY_PASSWORD=yes
volumes:
- /path/to/redis-persistence:/bitnami
airflow-worker:
image: bitnami/airflow-worker:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_LOAD_EXAMPLES=yes
airflow-scheduler:
image: bitnami/airflow-scheduler:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_LOAD_EXAMPLES=yes
airflow:
image: bitnami/airflow:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_PASSWORD=bitnami123
- AIRFLOW_USERNAME=user
- AIRFLOW_EMAIL=user@example.com
ports:
- '8080:8080'
Mount host directories as data volumes using the Docker command line
Create a network (if it does not exist)
docker network create airflow-tier
Create the PostgreSQL container with host volumes
docker run -d --name postgresql \
-e POSTGRESQL_USERNAME=bn_airflow \
-e POSTGRESQL_PASSWORD=bitnami1 \
-e POSTGRESQL_DATABASE=bitnami_airflow \
--net airflow-tier \
--volume /path/to/postgresql-persistence:/bitnami \
bitnami/postgresql:latest
Create the Redis(R) container with host volumes
docker run -d --name redis \
-e ALLOW_EMPTY_PASSWORD=yes \
--net airflow-tier \
--volume /path/to/redis-persistence:/bitnami \
bitnami/redis:latest
Create the Airflow container
docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e AIRFLOW_EMAIL=user@example.com \
--net airflow-tier \
bitnami/airflow:latest
Create the Airflow Scheduler container
docker run -d --name airflow-scheduler \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_LOAD_EXAMPLES=yes \
--net airflow-tier \
bitnami/airflow-scheduler:latest
Create the Airflow Worker container
docker run -d --name airflow-worker \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
--net airflow-tier \
bitnami/airflow-worker:latest
This container supports the installation of additional python modules at start-up time. In order to do that, you can mount a requirements.txt
file with your specific needs under the path /bitnami/python/requirements.txt
.
Customizable environment variables
Name | Description | Default Value |
---|---|---|
AIRFLOW_EXECUTOR | Airflow executor. | SequentialExecutor |
AIRFLOW_RAW_FERNET_KEY | Airflow raw/unencoded Fernet key | nil |
AIRFLOW_FERNET_KEY | Airflow Fernet key | nil |
AIRFLOW_SECRET_KEY | Airflow Secret key | nil |
AIRFLOW_FORCE_OVERWRITE_CONF_FILE | Force the airflow.cfg config file generation. | no |
AIRFLOW_WEBSERVER_HOST | Airflow webserver host | 127.0.0.1 |
AIRFLOW_WEBSERVER_PORT_NUMBER | Airflow webserver port. | 8080 |
AIRFLOW_HOSTNAME_CALLABLE | Method to obtain the hostname. | nil |
AIRFLOW_QUEUE | A queue for the worker to pull tasks from. | nil |
AIRFLOW_DATABASE_HOST | Hostname for PostgreSQL server. | postgresql |
AIRFLOW_DATABASE_PORT_NUMBER | Port used by PostgreSQL server. | 5432 |
AIRFLOW_DATABASE_NAME | Database name that Airflow will use to connect with the database. | bitnami_airflow |
AIRFLOW_DATABASE_USERNAME | Database user that Airflow will use to connect with the database. | bn_airflow |
AIRFLOW_DATABASE_PASSWORD | Database password that Airflow will use to connect with the database. | nil |
AIRFLOW_DATABASE_USE_SSL | Set to yes if the database is using SSL. | no |
AIRFLOW_REDIS_USE_SSL | Set to yes if Redis(R) uses SSL. | no |
REDIS_HOST | Hostname for Redis(R) server. | redis |
REDIS_PORT_NUMBER | Port used by Redis(R) server. | 6379 |
REDIS_USER | User that Airflow will use to connect with Redis(R). | nil |
REDIS_PASSWORD | Password that Airflow will use to connect with Redis(R). | nil |
REDIS_DATABASE | Name of the Redis(R) database. | 1 |
Read-only environment variables
Name | Description | Value |
---|---|---|
AIRFLOW_BASE_DIR | Airflow installation directory. | ${BITNAMI_ROOT_DIR}/airflow |
AIRFLOW_HOME | Airflow home directory. | ${AIRFLOW_BASE_DIR} |
AIRFLOW_BIN_DIR | Airflow directory for binary executables. | ${AIRFLOW_BASE_DIR}/venv/bin |
AIRFLOW_LOGS_DIR | Airflow logs directory. | ${AIRFLOW_BASE_DIR}/logs |
AIRFLOW_LOG_FILE | Airflow logs directory. | ${AIRFLOW_LOGS_DIR}/airflow-worker.log |
AIRFLOW_CONF_FILE | Airflow configuration file. | ${AIRFLOW_BASE_DIR}/airflow.cfg |
AIRFLOW_TMP_DIR | Airflow directory temporary files. | ${AIRFLOW_BASE_DIR}/tmp |
AIRFLOW_PID_FILE | Path to the Airflow PID file. | ${AIRFLOW_TMP_DIR}/airflow-worker.pid |
AIRFLOW_DAGS_DIR | Airflow data to be persisted. | ${AIRFLOW_BASE_DIR}/dags |
AIRFLOW_DAEMON_USER | Airflow system user. | airflow |
AIRFLOW_DAEMON_GROUP | Airflow system group. | airflow |
In addition to the previous environment variables, all the parameters from the configuration file can be overwritten by using environment variables with this format:
AIRFLOW__{SECTION}__{KEY}
. Note the double underscores.
Specifying Environment variables using Docker Compose
version: '2'
services:
airflow:
image: bitnami/airflow:latest
environment:
- AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
- AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08=
- AIRFLOW_EXECUTOR=CeleryExecutor
- AIRFLOW_DATABASE_NAME=bitnami_airflow
- AIRFLOW_DATABASE_USERNAME=bn_airflow
- AIRFLOW_DATABASE_PASSWORD=bitnami1
- AIRFLOW_PASSWORD=bitnami123
- AIRFLOW_USERNAME=user
- AIRFLOW_EMAIL=user@example.com
Specifying Environment variables on the Docker command line
docker run -d --name airflow -p 8080:8080 \
-e AIRFLOW_FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= \
-e AIRFLOW_SECRET_KEY=a25mQ1FHTUh3MnFRSk5KMEIyVVU2YmN0VGRyYTVXY08= \
-e AIRFLOW_EXECUTOR=CeleryExecutor \
-e AIRFLOW_DATABASE_NAME=bitnami_airflow \
-e AIRFLOW_DATABASE_USERNAME=bn_airflow \
-e AIRFLOW_DATABASE_PASSWORD=bitnami1 \
-e AIRFLOW_PASSWORD=bitnami123 \
-e AIRFLOW_USERNAME=user \
-e AIRFLOW_EMAIL=user@example.com \
bitnami/airflow:latest
docker-compose.yaml
file has been removed, as it was solely intended for internal testing purposes.We'd love for you to contribute to this Docker image. You can request new features by creating an issue or submitting a pull request with your contribution.
If you encountered a problem running this container, you can file an issue. For us to provide better support, be sure to fill the issue template.
Copyright © 2024 Broadcom. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.