This repository provides a Dockerised Kafka instance (based on Confluent Platform 3.0.1/Scala 2.11) for development and testing use.
NOTE: This image is NOT intended for use in production. Horrible things will probably happen if you attempt to do so.
Running Zookeeper and Kafka and configuring them to work together is a pain. This image runs both Zookeeper and Kafka together in a single container (managed by supervisord), pre-configured to work together out of the box.
By default Kafka will be accessible on
localhost:9193 so long as the port is exposed at runtime:
$ docker run -ti -p 9092:9092 -p 9093:9093 -p 9193:9193 paddycarey/kafka
Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default, each line will be sent as a separate message. You can use this client to test that Kafka is working. Run the producer and then type a few messages into the console to send to the server:
$ kafka-console-producer.sh --broker-list localhost:9092 --topic "test" This is a message This is another message
Kafka also has a command line consumer that will dump out messages to standard output.
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic "test" --from-beginning This is a message This is another message
If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal.
Accessing kafka on something other than localhost
This image is configured so that it can be accessed on
localhost:9092 for a PLAIN connection or
localhost:9093 for a SSL connection or
localhost:9193 for SSL + SASL. If you need to access Kafka using a different hostname you can set the
KAFKA_ADVERTISED_HOST environment variable.
$ docker run -ti -e "KAFKA_ADVERTISED_HOST=somehostname" -p 9092:9092 -p 9093:9093 -p 9193:9193 paddycarey/kafka
Or if using Docker compose, your
docker-compose.yml might look something like:
kafka: image: paddycarey/kafka # or whatever your built ports: - "9092:9092" - "9093:9093" - "9193:9193" environment: KAFKA_ADVERTISED_HOST: kafka consumer: build: . links: - kafka command: "python -u consumer.py"
This image stores log data in
/var/lib/kafka. You can use a Docker volume to persist this directory beyond the lifetime of a single container.
$ docker run -ti -v `pwd`/.data/:/var/lib/kafka -p 9092:9092 -p 9093:9093 -p 9193:9193 paddycarey/kafka
Building the image
To build the Docker image:
$ docker build -t paddycarey/kafka .
Note: Advanced users can modify the configuration files in
config/kafka/ if necessary to change the behaviour of Kafka or Zookeeper.