Public | Automated Build

Last pushed: 13 days ago
Short Description
Multi-Broker Apache Kafka Image
Full Description



kafka-docker

Dockerfile for Apache Kafka

The image is available directly from Docker Hub

##Pre-Requisites

  • install docker-compose https://docs.docker.com/compose/install/
  • modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)
  • if you want to customise any Kafka parameters, simply add them as environment variables in docker-compose.yml, e.g. in order to increase the message.max.bytes parameter set the environment to KAFKA_MESSAGE_MAX_BYTES: 2000000. To turn off automatic topic creation set KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

##Usage

Start a cluster:

  • docker-compose up -d

Add more brokers:

  • docker-compose scale kafka=3

Destroy a cluster:

  • docker-compose stop

##Note

The default docker-compose.yml should be seen as a starting point. By default each broker will get a new port number and broker id on restart. Depending on your use case this might not be desirable. If you need to use specific ports and broker ids, modify the docker-compose configuration accordingly, e.g. docker-compose-single-broker.yml:

  • docker-compose -f docker-compose-single-broker.yml up

##Broker IDs

If you don't specify a broker id in your docker-compose file, it will automatically be generated (see https://issues.apache.org/jira/browse/KAFKA-1070. This allows scaling up and down. In this case it is recommended to use the --no-recreate option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.

##Automatically create topics

If you want to have kafka-docker automatically create topics in Kafka during
creation, a KAFKA_CREATE_TOPICS environment variable can be
added in docker-compose.yml.

Here is an example snippet from docker-compose.yml:

    environment:
      KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"

Topic 1 will have 1 partition and 3 replicas, Topic 2 will have 1 partition, 1 replica and a cleanup.policy set to compact.

##Advertised hostname

You can configure the advertised hostname in different ways

  1. explicitly, using KAFKA_ADVERTISED_HOST_NAME
  2. via a command, using HOSTNAME_COMMAND, e.g. HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"

When using commands, make sure you review the "Variable Substitution" section in https://docs.docker.com/compose/compose-file/

If KAFKA_ADVERTISED_HOST_NAME is specified, it takes presendence over HOSTNAME_COMMAND

For AWS deployment, you can use the Metadata service to get the container host's IP:

HOSTNAME_COMMAND=wget -t3 -T2 -qO-  http://169.254.169.254/latest/meta-data/local-ipv4

Reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

JMX

For monitoring purposes you may wish to configure JMX. Additional to the standard JMX parameters, problems could arise from the underlying RMI protocol used to connect

  • java.rmi.server.hostname - interface to bind listening port
  • com.sun.management.jmxremote.rmi.port - The port to service RMI requests

For example, to connect to a kafka running locally (assumes exposing port 1099)

  KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
  JMX_PORT: 1099

Jconsole can now connect at jconsole 192.168.99.100:1099

##Tutorial

http://wurstmeister.github.io/kafka-docker/

Docker Pull Command
Owner
wurstmeister
Source Repository

Comments (17)
abhijitsarkar
2 months ago

The zookeeper image you're using requires login. Make it available without log in so that Kafka doesn't fail.

docker-compose -f docker-compose-single-broker.yml up
Creating network "kafka_default" with the default driver
Pulling zookeeper (wurstmeister/zookeeper:latest)...
ERROR: Get https://registry-1.docker.io/v2/wurstmeister/zookeeper/manifests/latest: unauthorized: incorrect username or password
milossimic
3 months ago

Hi, is there an easy way to send and receive messages using java with this setup?
I have tried, but i get error that kafka tries 3x times to send messages and failed :|.

tnx

blademainer
5 months ago

shit happend...
Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory

elonliu
6 months ago

hi man~ i am faced with some problem when use your project.The error code is :
kafka_1 | [2016-09-26 07:42:21,655] ERROR [KafkaApi-1002] Error when handling request {topics=[__consumer_offsets]} (kafka.server.KafkaApis)
kafka_1 | kafka.admin.AdminOperationException: replication factor: 3 larger than available brokers: 1
kafka_1 | at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
kafka_1 | at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:403)
kafka_1 | at kafka.server.KafkaApis.kafka$server$KafkaApis$$createTopic(KafkaApis.scala:629)
kafka_1 | at kafka.server.KafkaApis.kafka$server$KafkaApis$$createGroupMetadataTopic(KafkaApis.scala:651)
kafka_1 | at kafka.server.KafkaApis$$anonfun$29.apply(KafkaApis.scala:668)
kafka_1 | at kafka.server.KafkaApis$$anonfun$29.apply(KafkaApis.scala:666)
kafka_1 | at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
kafka_1 | at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
kafka_1 | at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
kafka_1 | at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
kafka_1 | at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
kafka_1 | at scala.collection.SetLike$class.map(SetLike.scala:92)
kafka_1 | at scala.collection.AbstractSet.map(Set.scala:47)
kafka_1 | at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:666)
kafka_1 | at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:727)
kafka_1 | at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
kafka_1 | at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
kafka_1 | at java.lang.Thread.run(Thread.java:745)

chanderg
8 months ago

Thanks for the image(s).

I had some issues, so I created an alternative approach, https://github.com/ChanderG/kafka-docker.

nangsaigon
8 months ago

Hi. Thanks for your Docker image. I'd like to have an idea about the port of cluster. From Apache Kafka 0.9, Kafka doesn't rely on Zookeeper port as the center point anymore. Rather than that, users can communicate with broker IPs, ports. So, in case we create many Kafka brokers, we should be able to define the ports for them exactly. What do you think?

belun
10 months ago

how do i find docker-compose.yaml (on windows) ?

jackeylee
10 months ago

Question: How to run this behind a proxy?

Unfortunately my server is behind a proxy and the "RUN apk add" command in Dockerfile could not execute since it could not fetch the APKINDEX.tar.gz file.

donvikram
10 months ago

Hi, can you explain please how I can add brokers with scale?

With this compose file:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    links:
      - zookeeper:zk
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.43
      KAFKA_CREATE_TOPICS: "FIRST_TOPIC:1:2,SECOND_TOPIC:2:3"
      KAFKA_ZOOKEEPER_CONNECT: zk:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

If i try docker-compose scale kafka=2 I get this error:

WARNING: The "kafka" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating and starting dockerlab_kafka_2 ... error
zlcolin
a year ago

Hi guy, an error found, any comments please? thanks
hs-zk | 2016-03-31 04:00:35,212 [myid:] - INFO [main:ZooKeeperServer@773] - maxSessionTimeout set to -1
hs-kafka | Cannot connect to the Docker daemon. Is the docker daemon running on this host?
hs-zk | 2016-03-31 04:00:35,227 [myid:] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
hs-kafka | [2016-03-31 04:00:36,053] FATAL (kafka.Kafka$)
hs-kafka | org.apache.kafka.common.config.ConfigException: Invalid value for configuration advertised.port: Not a number of type INT
hs-kafka | at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:253)
hs-kafka | at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:145)
hs-kafka | at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:49)
hs-kafka | at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:56)
hs-kafka | at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:702)
hs-kafka | at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:691)
hs-kafka | at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
hs-kafka | at kafka.Kafka$.main(Kafka.scala:58)
hs-kafka | at kafka.Kafka.main(Kafka.scala)
hs-km | [warn] o.a.c.r.ExponentialBackoffRetry - maxRetries too large (100). Pinning to 29
hs-km | [info] k.m.a.KafkaManagerActor - Starting curator...
hs-km | [info] o.a.z.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
hs-km | [info] o.a.z.ZooKeeper - Client environment:host.name=58d050f91bbd
hs-km | [info] o.a.z.ZooKeeper - Client environment:java.version=1.8.0_73
hs-km | [info] o.a.z.ZooKeeper - Client environment:java.vendor=Oracle Corporation