Public Repository

Last pushed: 4 months ago
Short Description
Kafka with dynamic configuration, auto hostname detection and topic creaton, and ZK checks
Full Description

monasca/kafka Dockerfile

This image runs an instance of Apache Kafka optimized for Docker and
Kubernetes environments.

Sources: kafka · Dockerfile · monasca-docker


Images in this repository are tagged as follows:

  • [kafka version]-[scala version], e.g.
  • latest: the latest version recommended for use with other Monasca
    components. Currently, this is
  • master: a development / testing build not necessarily intended for
    production use


Kafka requires a running instance of Zookeeper. The library zookeeper image
is recommended:

docker run --name zookeeper -d zookeeper:3.3

If Zookeeper is accessible at zookeeper:2181, no additional options need to be

docker run --name kafka --link zookeeper monasca/kafka:latest

Kafka will be running on port 9092 by default. If JMX is enabled, that port
will be available as well (7203 by default). These ports can be exposed as
necessary (-p 9092:9092 or similar), but keep in mind that forwarding ports
will impact connectivity unless KAFKA_ADVERTISED_HOST_NAME is set correctly as
described below.


Several parameters can be specified using environment variables:

Variable Default Description
ZOOKEEPER_CONNECTION_STRING zookeeper:2181 Comma-separated list of ZK hosts
ZOOKEEPER_WAIT true If true, wait for zookeeper to become ready
ZOOKEEPER_WAIT_TIMEOUT 3 Connection timeout for ZK wait loop
ZOOKEEPER_WAIT_DELAY 10 Seconds to wait between connection attempts
ZOOKEEPER_WAIT_RETRIES 20 Password for given user
ZOOKEEPER_CHROOT unset Optional ZK chroot / path prefix
KAFKA_HOSTNAME_FROM_IP true If true, set advertised hostname to container IP
KAFKA_ADVERTISED_HOST_NAME from IP If set, use value as advertised hostname
KAFKA_BROKER_ID -1 Unique Kafka broker ID, -1 for auto
KAFKA_LISTEN_PORT 9092 Port for Kafka to listen on
KAFKA_ADVERTISED_PORT $KAFKA_LISTEN_PORT Kafka port advertised to clients
KAFKA_CONTROLLED_SHUTDOWN_ENABLE true If true, enable controlled shutdown
KAFKA_JMX unset If true, expose JMX metrics over TCP
KAFKA_JMX_PORT 7203 Port to expose JMX metrics
KAFKA_JMX_OPTS no SSL/auth, etc Override default opts
SERVER_LOG_LEVEL INFO Log Level for server
REQUEST_LOG_LEVEL WARN Log Level for request logging
CONTROLLER_LOG_LEVEL INFO Log Level for controller
LOG_CLEANER_LOG_LEVEL INFO Log Level for log cleaner
STATE_CHANGE_LOG_LEVEL INFO Log Level for state changes
AUTHORIZER_LOG_LEVEL WARN Log Level for the authorizer
GC_LOG_ENABLED False If True, JVM garbage collection log enabled
KAFKA_STACK_SIZE 1024k JVM stack size
LOG_RETENTION_HOURS 4 Number of hours to keep a log file
LOG_ROLL_MS 900000 Number of ms before a new log segment is rolled out
STAY_ALIVE_ON_FAILURE false If true, container stays alive for 2 hours after kafka exits

Log Files

Multiple Kafka log files are written to stdout for the container. They can be distinguished via
the logfile attribute in each message. For example:

kafka_1                 | [2017-06-15 06:32:09,572] INFO logfile=server.log Loading logs. (kafka.log.LogManager)

If GC_LOG_ENABLED is set to True, the JVM Garbage Collection log will be written within the
container at /kafka/logs/kafkaServer-gc.log. It can't be redirected to stdout.

Docker Pull Command