ueisele/apache-kafka-connect-base repository overview

⁠Container Image for Apache Kafka Connect (Base)

Container image for running the Open Source version of Apache Kafka Connect⁠ in distributed mode.

The Kafka distribution included in the Container image is built directly from source⁠. It is a base image and therefore has no additional Cli tools and Kafka Connect plugins installed.

The Container images are available on Docker Hub repository ueisele/apache-kafka-connect-base⁠, and the source files for the images are available on GitLab repository ueisele/kafka-images⁠.

⁠Most Recent Tags

Most recent tags for RELEASE builds:

• 3.5.0, 3.5.0-zulu20, 3.5.0-zulu20.0.1, 3.5.0-zulu20-alma9.2, 3.5.0-zulu20.0.1-alma9.2-20230512
• 3.4.1, 3.4.1-zulu17, 3.4.1-zulu17.0.7, 3.4.1-zulu17-alma9.2, 3.4.1-zulu17.0.7-alma9.2-20230512

Most recent tags for SNAPSHOT builds:

• 3.6.0-SNAPSHOT, 3.6.0-SNAPSHOT-zulu20, 3.6.0-SNAPSHOT-zulu20.0.1, 3.6.0-SNAPSHOT-zulu20-alma9.2, 3.6.0-SNAPSHOT-zulu20.0.1-alma9.2-20230512

Additionally, a tag with the associated Git-Sha of the built Apache Kafka distribution is always published as well, e.g. ueisele/apache-kafka-connect-base:3.6.0-SNAPSHOT-g09e8adb.

⁠Image

The Container images are based on ueisele/zulu-openjdk-micro⁠ with JRE installed (e.g. 20-jre).

The OpenJDK image in turn is based on AlmaLinux 9 Micro⁠.

As OpenJDK Azul Zulu⁠ is used. Azul Zulu builds of OpenJDK are fully tested and TCK compliant builds of OpenJDK.

⁠Quick Start

In the following section you find some simple examples to run Apache Kafka Connect.

First create a Container network:

podman network create quickstart-kafka-connect

Now, start a single Kafka instance:

podman run -d --name kafka --net quickstart-kafka-connect -p 9092:9092 ueisele/apache-kafka-server-standalone:3.5.0

In order to run Apache Kafka Connect with a single instance run the following command:

podman run -d --name kafka-connect --net quickstart-kafka-connect -p 8083:8083 \
    -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
    -e CONNECT_REST_ADVERTISED_HOST_NAME=localhost \
    -e CONNECT_REST_PORT=8083 \
    -e CONNECT_GROUP_ID=quickstart-kafka-connect \
    -e CONNECT_CONFIG_STORAGE_TOPIC=quickstart-kafka-connect-config \
    -e CONNECT_OFFSET_STORAGE_TOPIC=quickstart-kafka-connect-offsets \
    -e CONNECT_STATUS_STORAGE_TOPIC=quickstart-kafka-connect-status \
    -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=false \
    -e CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=false \
    ueisele/apache-kafka-connect-base:3.5.0

You find additional examples in examples/connect-standalone/⁠:

• examples/connect-standalone/file-source/compose.yaml⁠
• examples/connect-standalone/http-source-plugin-install/compose.yaml⁠

⁠Configuration

For the Apache Kafka Connect (ueisele/apache-kafka-connect-base⁠) image, convert the Apache Kafka Connect configuration properties⁠ as below and use them as environment variables:

• Prefix with CONNECT_.
• Convert to upper-case.
• Replace a period (.) with a single underscore (_).
• Replace a dash (-) with double underscores (__).
• Replace an underscore () with triple underscores (__).

The configuration is fully compatible with the Confluent Docker images⁠.

The configuration mechanism supports Go Template⁠ for environment variable values. The templating is done by godub⁠ and therefore provides its template functions⁠.

Example which uses ipAddress function to determin the IPv4 address of the first network interface:

CONNECT_REST_ADVERTISED_HOST_NAME="{{ ipAddress \"prefer\" \"ipv4\" 0 }}"

⁠Required Configuration

The minimum required worker configuration is:

• CONNECT_BOOTSTRAP_SERVERS which defines the the Kafka bootstrap servers
• CONNECT_KEY_CONVERTER and CONNECT_VALUE_CONVERTER which define the converters used for key and value.
• CONNECT_GROUP_ID which identifies the Connect cluster group this Worker belongs to.
• CONNECT_CONFIG_STORAGE_TOPIC, CONNECT_OFFSET_STORAGE_TOPIC and CONNECT_STATUS_STORAGE_TOPIC which define the names of the topics where connector tasks, configuration, offsets and status updates are stored. This names must be unique per Connect cluster.
• CONNECT_REST_ADVERTISED_HOST_NAME which defines the hostname that will be given out to other Workers to connect to. You should set this to a value that is resolvable by all containers.

⁠Logging

The logging configuration can be adjusted with the following environment variables:

• CONNECT_LOG4J_PATTERN sets the logging pattern (default: [%d] (%t) %p %m (%c)%n)
• CONNECT_LOG4J_ROOT_LOGLEVEL sets the root log level (default: INFO)
• CONNECT_LOG4J_LOGGERS is a comma separated list of logger and log level key-value pairs (default: org.reflections=ERROR,org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR)

⁠JMX

Remote JMX can be enabled with the following environment variables:

KAFKA_JMX_PORT=6001
KAFKA_JMX_HOSTNAME=localhost

⁠Debugging

In order to debug Kafka Connect, set the following environment variable:

KAFKA_DEBUG=y

In addition you can configure the behavior with the following environment variables:

JAVA_DEBUG_PORT=5005
DEBUG_SUSPEND_FLAG=y

⁠Kafka Connect Plugin Installation

You can install Kafka Connect plugins with multi stage builds. The connect/Containerfile⁠ can be used as an example.

If you want to be able to directly install plugins in the image, use ueisele/apache-kafka-connect-dev⁠ instead.

⁠Build

In order to create your own Container image for Apache Kafka Connect clone the ueisele/kafka-image⁠ Git repository and run the build command:

git clone https://gitlab.com/ueisele/kafka-images.git
cd kafka-images
connect-base/build.sh --build --tag 3.5.0 --openjdk-release 20

To create an image with a specific OpenJDK version use the following command:

connect-base/build.sh --build --tag 3.5.0 --openjdk-release 20 --openjdk-version 20.0.1

By default Apache Kafka 3.0.0 does not support Java 17. In order to build Apache Kafka 3.0.0 with Java 17, the Gradle configuration is patched with patch/3.0.0-openjdk17.patch⁠.

connect-base/build.sh --build --tag 3.0.0 --openjdk-release 17 --patch 3.0.0-openjdk17.patch

To build the most recent SNAPSHOT of Apache Kafka 3.5.0 with Java 17, run:

connect-base/build.sh --build --branch trunk --openjdk-release 17

⁠Build Options

The connect-base/build.sh script provides the following options:

Usage: connect-base/build.sh [--build] [--push] [--registry docker.io] [--user ueisele] [--archs amd64,arm64] [--github-repo apache/kafka] [--commit-sha 09e8adb] [--tag 3.5.0] [--branch trunk] [--pull-request 9999] [--openjdk-release 20] [--openjdk-version 20.0.1] [--patch 3.0.0-openjdk17.patch] [--cache-maven <m2-directory>] [--cache-gradle <gradle-directory>]

⁠License

This Container image is licensed under the Apache 2 license⁠.