ueisele/apache-kafka-connect-dev repository overview

⁠Container Image for Apache Kafka Connect

Container image for running the Open Source version of Apache Kafka Connect⁠ in distributed mode.

The Kafka distribution included in the Container image is built directly from source⁠.

The Kafka Connect Container image is based on ueisele/apache-kafka-connect-base⁠. It has the Confluent Hub Cli, as well as additional Kafka Connect plugins pre-installed.

The Container images are available on Docker Hub repository ueisele/apache-kafka-connect-dev⁠, and the source files for the images are available on GitLab repository ueisele/kafka-images⁠.

The Apache Kafka Connect Container image supports installation of Kafka Connect plugins like connectors during startup with multiple methods.

⁠Most Recent Tags

Most recent tags for RELEASE builds:

• 3.5.0, 3.5.0-zulu20, 3.5.0-zulu20.0.1, 3.5.0-zulu20-alma9.2, 3.5.0-zulu20.0.1-alma9.2-20230512
• 3.4.1, 3.4.1-zulu17, 3.4.1-zulu17.0.7, 3.4.1-zulu17-alma9.2, 3.4.1-zulu17.0.7-alma9.2-20230512

Most recent tags for SNAPSHOT builds:

• 3.6.0-SNAPSHOT, 3.6.0-SNAPSHOT-zulu20, 3.6.0-SNAPSHOT-zulu20.0.1, 3.6.0-SNAPSHOT-zulu20-alma9.2, 3.6.0-SNAPSHOT-zulu20.0.1-alma9.2-20230512

Additionally, a tag with the associated Git-Sha of the built Apache Kafka distribution is always published as well, e.g. ueisele/apache-kafka-connect:3.6.0-SNAPSHOT-g09e8adb.

⁠Image

The Container images are based on ueisele/zulu-openjdk-micro⁠ with JRE installed (e.g. 20-jre).

The OpenJDK image in turn is based on AlmaLinux 9 Micro⁠.

As OpenJDK Azul Zulu⁠ is used. Azul Zulu builds of OpenJDK are fully tested and TCK compliant builds of OpenJDK.

⁠Quick Start

In the following section you find some simple examples to run Apache Kafka Connect.

First create a Container network:

podman network create quickstart-kafka-connect

Now, start a single Kafka instance:

podman run -d --name kafka --net quickstart-kafka-connect -p 9092:9092 ueisele/apache-kafka-server-standalone:3.5.0

In order to run Apache Kafka Connect with a single instance run the following command:

podman run -d --name kafka-connect --net quickstart-kafka-connect -p 8083:8083 \
    -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
    -e CONNECT_REST_ADVERTISED_HOST_NAME=localhost \
    -e CONNECT_REST_PORT=8083 \
    -e CONNECT_GROUP_ID=quickstart-kafka-connect \
    -e CONNECT_CONFIG_STORAGE_TOPIC=quickstart-kafka-connect-config \
    -e CONNECT_OFFSET_STORAGE_TOPIC=quickstart-kafka-connect-offsets \
    -e CONNECT_STATUS_STORAGE_TOPIC=quickstart-kafka-connect-status \
    -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \
    -e CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=false \
    -e CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=false \
    ueisele/apache-kafka-connect:3.5.0

You find additional examples in examples/connect-standalone/⁠:

• examples/connect-standalone/datagen-plugin-install/compose.yaml⁠
• examples/connect-standalone/http-source-plugin-install/compose.yaml⁠

⁠Configuration

For the Apache Kafka Connect (ueisele/apache-kafka-connect⁠) image, convert the Apache Kafka Connect configuration properties⁠ as below and use them as environment variables:

• Prefix with CONNECT_.
• Convert to upper-case.
• Replace a period (.) with a single underscore (_).
• Replace a dash (-) with double underscores (__).
• Replace an underscore () with triple underscores (__).

The configuration is fully compatible with the Confluent Docker images⁠.

The configuration mechanism supports Go Template⁠ for environment variable values. The templating is done by godub⁠ and therefore provides its template functions⁠.

Example which uses ipAddress function to determin the IPv4 address of the first network interface:

CONNECT_REST_ADVERTISED_HOST_NAME="{{ ipAddress \"prefer\" \"ipv4\" 0 }}"

⁠Required Configuration

The minimum required worker configuration is:

• CONNECT_BOOTSTRAP_SERVERS which defines the the Kafka bootstrap servers
• CONNECT_KEY_CONVERTER and CONNECT_VALUE_CONVERTER which define the converters used for key and value.
• CONNECT_GROUP_ID which identifies the Connect cluster group this Worker belongs to.
• CONNECT_CONFIG_STORAGE_TOPIC, CONNECT_OFFSET_STORAGE_TOPIC and CONNECT_STATUS_STORAGE_TOPIC which define the names of the topics where connector tasks, configuration, offsets and status updates are stored. This names must be unique per Connect cluster.
• CONNECT_REST_ADVERTISED_HOST_NAME which defines the hostname that will be given out to other Workers to connect to. You should set this to a value that is resolvable by all containers.

⁠Logging

The logging configuration can be adjusted with the following environment variables:

• CONNECT_LOG4J_PATTERN sets the logging pattern (default: [%d] (%t) %p %m (%c)%n)
• CONNECT_LOG4J_ROOT_LOGLEVEL sets the root log level (default: INFO)
• CONNECT_LOG4J_LOGGERS is a comma separated list of logger and log level key-value pairs (default: org.reflections=ERROR,org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR)

⁠JMX

Remote JMX can be enabled with the following environment variables:

KAFKA_JMX_PORT=6001
KAFKA_JMX_HOSTNAME=localhost

⁠Debugging

In order to debug Kafka Connect, set the following environment variable:

KAFKA_DEBUG=y

In addition you can configure the behavior with the following environment variables:

JAVA_DEBUG_PORT=5005
DEBUG_SUSPEND_FLAG=y

⁠Kafka Connect Plugin Installation

The Apache Kafka Connect Container image supports installation of Kafka Connect plugins like connectors during startup with multiple methods.

Define a comma separated list of plugins which should be installed via Confluent Hub.

PLUGIN_INSTALL_CONFLUENT_HUB_IDS: |
    confluentinc/kafka-connect-jdbc:latest
    confluentinc/kafka-connect-http:latest

Define a comma separated list of plugin Urls. Supported are *.zip, *.tar*, *.tgz and *.jar files.

PLUGIN_INSTALL_URLS: |
    https://github.com/castorm/kafka-connect-http/releases/download/v0.8.11/castorm-kafka-connect-http-0.8.11.zip
    https://github.com/RedHatInsights/expandjsonsmt/releases/download/0.0.7/kafka-connect-smt-expandjsonsmt-0.0.7.tar.gz

Define a comma separated list of 'path=url' pairs, to download additional libraries. Supported are *.zip, *.tar*, *.tgz and *.jar files.

PLUGIN_INSTALL_LIB_URLS: |
    confluentinc-kafka-connect-jdbc/lib=https://dlm.mariadb.com/1496775/Connectors/java/connector-java-2.7.2/mariadb-java-client-2.7.2.jar
    confluentinc-kafka-connect-avro-converter/lib=https://repo1.maven.org/maven2/com/google/guava/guava/30.1.1-jre/guava-30.1.1-jre.jar

⁠Pre-installed Kafka Connect Plugins

This Kafka Connect image has the Confluent converters for Avro, Protobuf and JSON Schema already pre-installed to simplify usage of Confluent Schema Registry.

• confluentinc/kafka-connect-avro-converter:7.4.0⁠
• confluentinc/kafka-connect-protobuf-converter:7.4.0⁠
• confluentinc/kafka-connect-json-schema-converter:7.4.0⁠

This Kafka Connect image has the Confluent Connect SMTs already pre-installed:

• confluentinc/connect-transforms:1.4.3⁠

⁠Build

In order to create your own Container image for Apache Kafka Connect clone the ueisele/kafka-image⁠ Git repository and run the build command:

git clone https://gitlab.com/ueisele/kafka-images.git
cd kafka-images
connect/build.sh --build --tag 3.5.0 --openjdk-release 20

To create an image with a specific OpenJDK version use the following command:

connect/build.sh --build --tag 3.5.0 --openjdk-release 20 --openjdk-version 20.0.1

By default Apache Kafka 3.0.0 does not support Java 17. In order to build Apache Kafka 3.0.0 with Java 17, the Gradle configuration is patched with patch/3.0.0-openjdk17.patch⁠.

connect/build.sh --build --tag 3.0.0 --openjdk-release 17 --patch 3.0.0-openjdk17.patch

To build the most recent SNAPSHOT of Apache Kafka 3.5.0 with Java 17, run:

connect/build.sh --build --branch trunk --openjdk-release 17

⁠Build Options

The connect/build.sh script provides the following options:

Usage: connect/build.sh [--build] [--push] [--registry docker.io] [--user ueisele] [--archs amd64,arm64] [--github-repo apache/kafka] [--commit-sha 09e8adb] [--tag 3.5.0] [--branch trunk] [--pull-request 9999] [--openjdk-release 20] [--openjdk-version 20.0.1] [--patch 3.0.0-openjdk17.patch]

⁠License

This Container image is licensed under the Apache 2 license⁠.