Sign inSign up

cloudurable/kafka-image

By cloudurable

•Updated about 4 years ago

Kafka Image

Image
0

523

cloudurable/kafka-image repository overview

⁠What is Apache Kafka?

Apache Kafka provides ...

⁠Run

Run the 9th server / broker in a cluster.

docker run  -d \
    -e KAFKA_BROKER_ID=9  \
    -e KAFKA_CLUSTER_MODE=auto \
    -e KAFKA_ZOOKEEPER_CONNECT=zk0.cloudurable.com::2181,zk1.cloudurable.com::2181,zk2.cloudurable.com::2181 \
    cloudurable/kafka-image

Run standalone for testing.

docker run  -d  -e KAFKA_CLUSTER_MODE=standalone \
      --restart always --net=host \
      cloudurable/kafka-image

Debug mode

docker run  -d  -e KAFKA_CLUSTER_MODE=standalone \
        -e KAFKA_DEBUG_MODE=true \
      --restart always --net=host \
      cloudurable/kafka-image

⁠Environment Variables

      "changes": [
          "ENV KAFKA_CLUSTER_MODE standalone",
          "ENV KAFKA_DEFAULT_REPLICATION 3",
          "ENV KAFKA_BROKER_ID 1",
          "ENV KAFKA_LOG_DIRS /opt/kafka/data",
          "ENV KAFKA_PORT 9092",
          "ENV KAFKA_LISTENERS ''",
          "ENV KAFKA_ADVERTISED_LISTENERS ''",
          "ENV KAFKA_NUM_NETWORK_THREADS 3",
          "ENV KAFKA_NUM_IO_THREADS 8",
          "ENV KAFKA_NUM_PARTITIONS 1",
          "ENV KAFKA_SOCKET_SEND_BUFFER_BYTES 102400",
          "ENV KAFKA_SOCKET_RECEIVE_BUFFER_BYTES 102400",
          "ENV KAFKA_SOCKET_REQUEST_MAX_BYTES 104857600",
          "ENV KAFKA_NUM_RECOVERY_THREADS_PER_DATA_DIR 1",
          "ENV KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR 1",
          "ENV KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR 1",
          "ENV KAFKA_TRANSACTION_STATE_LOG_MIN_ISR 1",
          "ENV KAFKA_LOG_RETENTION_HOURS 168",
          "ENV KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS 300000",
          "ENV KAFKA_ZOOKEEPER_CONNECT localhost:2181",
          "ENV KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT 6000",
          "ENV KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS 3000",
          "ENV KAFKA_LOG_RETENTION_BYTES ''",
          "ENTRYPOINT /opt/kafka/bin/run.sh"

        ]

⁠Example Docker Compose file to run an Ensemble of ZooKeeper servers (cluster)

version: '3'
services:
  zookeeper0:
    image: "cloudurable/zookeeper-image:0.3"
    restart: always
    ports:
     - "2181:2181"
    environment:
     - ENSEMBLE=zookeeper0,zookeeper1,zookeeper2
     - MY_ID=1
  zookeeper1:
    image: "cloudurable/zookeeper-image:0.3"
    restart: always
    ports:
     - "2182:2181"
    environment:
     - ENSEMBLE=zookeeper0,zookeeper1,zookeeper2
     - MY_ID=2
  zookeeper2:
    image: "cloudurable/zookeeper-image:0.3"
    restart: always
    ports:
     - "2183:2181"
    environment:
     - ENSEMBLE=zookeeper0,zookeeper1,zookeeper2
     - MY_ID=3
  kafka0:
    image: "cloudurable/kafka-image"
    restart: always
    depends_on:
      - zookeeper0
      - zookeeper1
      - zookeeper2
    ports:
     - "9092:9092"
    links:
     - zookeeper0
     - zookeeper1
     - zookeeper2
    environment:
     - KAFKA_ZOOKEEPER_CONNECT=zookeeper0:2181,zookeeper1:2181,zookeeper2:2181
     - KAFKA_BROKER_ID=1
     - KAFKA_CLUSTER_MODE=auto
     - KAFKA_LISTENERS=PLAINTEXT://kafka0:9092
     - KAFKA_DEFAULT_REPLICATION=1

This above starts Zookeeper in replicated mode⁠. Run docker-compose up\ and wait for it to initialize completely. Ports 2181, 2888 and 3888 will be exposed.

⁠Configuration

The Kafka config gets generated based on the environment variables that you specify by /opt/kafka/bin/run.sh using sed to edit the Kafka config file.

Kafka configuration is located in /opt/kafka/conf/server.properties.

To change the config, mount a new config file as a volume:

	$ docker run --name node0-zookeeper --restart always \
      -d -v $(pwd)/kafka.cfg:/opt/kafka/conf/server.properties \
      cloudurable/kafka-image

⁠Dev Notes

⁠Build base image

packer build base-java-packer.json

We build a base image which installs Java, updates the OS, etc. This is time consuming. We try to do it once.

⁠Run packer

packer build docker-packer.json

⁠Snoop around

docker run -it cloudurable/zookeeper-image:latest
## We added an entrypoint so now you have to do this
docker run -it --entrypoint "/bin/bash"  cloudurable/kafka-image

⁠Clean docker

docker ps -a | grep cloudurable/kafka-image | awk '{ print $1}' \
  | xargs docker rm

docker images | grep cloudurable/kafka-image | awk '{ print $3}' \
    | xargs docker rmi -f

⁠Push another tag

$ docker tag 3e67b5405fc4  cloudurable/kafka-image:0.7
$ docker push  cloudurable/kafka-image:0.7

Tag summary

Content type

Image

Digest

Size

417.6 MB

Last updated

about 4 years ago

docker pull cloudurable/kafka-image