Public | Automated Build

Last pushed: a year ago
Short Description
Docker image for creating a local multi-broker Kafka cluster
Full Description

Local Docker Kafka Cluster


Creates a Docker-based Kafka
cluster with docker-compose for local
development of Kafka-based apps.

Prerequisites

Works for Linux and Mac (with the latest Docker for Mac), and uses docker
compose file syntax version 2.

Also, the ports 2181, 9092, 9093, and 9094 (for a default three broker
setup) should be available.

Running

Bring up the cluster and specify that you want e.g. five brokers (the default
cluster size is three brokers) with

./cluster-up 5

This will create a docker-compose.yml file in the directory that will be used
by ./cluster-down do delete the cluster (i.e. this stops and removes all data).
If you do not want to delete data, use the docker-compose commands directly.

Connectivity

The cluster will contain one ZooKeeper node only accessible as localhost:2181.
The brokers will be reachable under localhost:9092,localhost:9093,... and
advertise Docker host IP to Kafka clients. As this is reachable from the host
and from the Docker network, it is possible to connect to the cluster both
from the host machine as well as from other Docker containers.

The Thing With the Docker Network and Kafka

Kafka Brokers advertise themselves to each other and clients via the Kafka
protocol, and announce hostname/IP and port. It is therefore necessary to have
fixed (distinct) ports, and have a hostname or IP that is resolvable or routable
from both the docker network (for broker-to-broker communication) and from the
docker host (for local development).

To achieve this for both Linux and Mac - in the face of
some verified existing issues
with Mac - the best way to go is to use the docker host IP.
Without that issue, it would be more sensible to use the docker container IPs.

Using the docker host IP as advertised broker address makes broker-to-broker
traffic pass through the docker host, but this overhead is considered an
acceptable price for a solution working both on Linux and Mac.

Docker Pull Command
Owner
andschroeder

Comments (0)