epwalsh/docker-celery repository overview

⁠docker-celery

This is a Docker image for Python Celery workers.

⁠Details

• Built on Alpine Linux for a super small image size.
• The Celery process is managed by Supervisor, and is pointed to the application through an environment variable.
• Other environment variables control the queue name, number of workers, and log level.
• Uses an external broker so the number of workers can be scaled up by simply scaling the number of containers.

⁠Quick start example

This will show you how to get the example application⁠ running.

In order to get started, you will need a message broker running, such as RabbitMQ. If you don't already have one, you can get a free RabbitMQ instance for development purposes through CloudAMQP⁠. It should only take a few minutes to set up.

Once you have one, take note of the AMQP URL, which will look something like this: amqp://****:*****@donkey.rmqp.cloudamqp.com/****.

Now, if you haven't already, clone this repository and pull the Docker image:

git clone https://github.com/epwalsh/docker-celery && cd docker-celery
docker pull epwalsh/celery

Next, cd into the example/ directory and create the environment file access.txt with the AMQP URL:

cd example
cp access.sample.txt access.txt

Edit the first line in access.txt so that BROKER_URL is set to your AMQP URL. Now build and run the image with

docker build -t YOUR-NAME/celery-test .
docker run --env-file=./access.txt --rm YOUR-NAME/celery-test

If everything works, the Celery worker should start and produce some messages like this:

[2018-05-26 21:33:29,024: INFO/MainProcess] Connected to amqp://****:**@donkey.rmq.cloudamqp.com:****/****
[2018-05-26 21:33:29,582: INFO/MainProcess] mingle: searching for neighbors
[2018-05-26 21:33:31,737: INFO/MainProcess] mingle: all alone
[2018-05-26 21:33:32,877: INFO/MainProcess] test_queue-worker@4e39494c2213 ready.

You can now send a task to your worker with a Python script like this:

from celery import Celery

celery_app = Celery("example_app", broker="amqp://****")  # `broker` should be your BROKER_URL from access.txt
celery_app.send_task("worker.tasks.hello_world", queue="test_queue")