ueisele/apache-kafka-connect-standalone-dev repository overview

⁠Container Image for Apache Kafka Connect Standalone

Container image for running the Open Source version of Apache Kafka Connect⁠ in standalone mode.

The Kafka distribution included in the Container image is built directly from source⁠.

The standalone Container image is based on ueisele/apache-kafka-connect⁠.

The Container images are available on Docker Hub repository ueisele/apache-kafka-connect-standalone-dev⁠, and the source files for the images are available on GitLab repository ueisele/kafka-images⁠.

The Apache Kafka Connect Container image supports installation of Kafka Connect plugins like connectors during startup with multiple methods.

IMPORTANT: Kafka Connect Standalone is not suited for most productive uses cases. It does not support scaling and offsets of external systems are just stored on a file. See also: https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-kafka-connect-workers/⁠

⁠Most Recent Tags

Most recent tags for RELEASE builds:

• 3.5.0, 3.5.0-zulu20, 3.5.0-zulu20.0.1, 3.5.0-zulu20-alma9.2, 3.5.0-zulu20.0.1-alma9.2-20230512
• 3.4.1, 3.4.1-zulu17, 3.4.1-zulu17.0.7, 3.4.1-zulu17-alma9.2, 3.4.1-zulu17.0.7-alma9.2-20230512

Most recent tags for SNAPSHOT builds:

• 3.6.0-SNAPSHOT, 3.6.0-SNAPSHOT-zulu20, 3.6.0-SNAPSHOT-zulu20.0.1, 3.6.0-SNAPSHOT-zulu20-alma9.2, 3.6.0-SNAPSHOT-zulu20.0.1-alma9.2-20230512

Additionally, a tag with the associated Git-Sha of the built Apache Kafka distribution is always published as well, e.g. ueisele/apache-kafka-connect-standalone:3.6.0-SNAPSHOT-g09e8adb.

⁠Image

The Container images are based on ueisele/zulu-openjdk-micro⁠ with JRE installed (e.g. 20-jre).

The OpenJDK image in turn is based on AlmaLinux 9 Micro⁠.

As OpenJDK Azul Zulu⁠ is used. Azul Zulu builds of OpenJDK are fully tested and TCK compliant builds of OpenJDK.

⁠Quick Start

In the following section you find some simple examples to run Apache Kafka Connect.

First create a Container network:

podman network create quickstart-kafka-connect-standalone

Now, start a single Kafka instance:

podman run -d --name kafka --net quickstart-kafka-connect-standalone -p 9092:9092 ueisele/apache-kafka-server-standalone:3.5.0

In order to run Apache Kafka Connect in standalone mode run the following command:

podman run -d --name kafka-connect-standalone \
    --net quickstart-kafka-connect-standalone -p 8083:8083 \
    -e PLUGIN_INSTALL_CONFLUENT_HUB_IDS=confluentinc/kafka-connect-datagen:latest \
    -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
    -e CONNECT_KEY_CONVERTER=org.apache.kafka.connect.storage.StringConverter \
    -e CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE="false" \
    -e CONNECT_OFFSET_FLUSH_INTERVAL_MS=5000 \
    -e CONNECTOR_NAME=datagen-source \
    -e CONNECTOR_CONNECTOR_CLASS=io.confluent.kafka.connect.datagen.DatagenConnector \
    -e CONNECTOR_TASKS_MAX=1 \
    -e CONNECTOR_KAFKA_TOPIC=connect-datagen-source \
    -e CONNECTOR_QUICKSTART=users \
    ueisele/apache-kafka-connect-standalone:3.5.0

Consume published messages:

podman run --rm -it --net quickstart-kafka-connect-standalone ueisele/apache-kafka-server-standalone:3.5.0 \
    kafka-console-consumer.sh \
        --bootstrap-server kafka:9092 \
        --topic connect-datagen-source \
        --from-beginning

You find additional examples in examples/connect-standalone/⁠:

• examples/connect-standalone/datagen-plugin-install/compose.yaml⁠
• examples/connect-standalone/http-source-plugin-install/compose.yaml⁠

⁠Configuration

For the Apache Kafka Connect (ueisele/apache-kafka-connect-standalone⁠) image, convert the Apache Kafka Connect configuration properties⁠ as below and use them as environment variables:

• Prefix with CONNECT_ for worker configuration.
• Prefix with CONNECTOR_ for connector configuration.
• Convert to upper-case.
• Replace a period (.) with a single underscore (_).
• Replace a dash (-) with double underscores (__).
• Replace an underscore () with triple underscores (__).

The configuration is fully compatible with the Confluent Docker images⁠.

The configuration mechanism supports Go Template⁠ for environment variable values. The templating is done by godub⁠ and therefore provides its template functions⁠.

⁠Required Worker Configuration

The minimum required worker configuration is CONNECT_BOOTSTRAP_SERVERS which defines the the Kafka bootstrap servers and CONNECT_KEY_CONVERTER and CONNECT_VALUE_CONVERTER which define the converters used for key and value.

CONNECT_BOOTSTRAP_SERVERS: kafka:9092
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.storage.StringConverter

⁠Required Connector Configuration

The minimum required configuration for a connector is CONNECTOR_NAME which defines the connector instance name and CONNECTOR_CONNECTOR_CLASS which defines the class implementing the connector.

CONNECTOR_NAME: file-source
CONNECTOR_CONNECTOR_CLASS: FileStreamSource

⁠Offset Storage

Kafka Connect standalone does maintain its offsets in a file. This file is by default located at /opt/apache/kafka/data/connect.offsets You can change the file name by setting the following configuration.

CONNECT_STANDALONE_OFFSET_STORAGE_FILE_FILENAME: file-source.offsets

In order to save the offset, you should always bind the /opt/apache/kafka/data/ directory as dedicated volume.

You can also specify the flush interval for the offsets. By default its one minute.

CONNECT_OFFSET_FLUSH_INTERVAL_MS: 5000

⁠Logging

The logging configuration can be adjusted with the following environment variables:

• CONNECT_LOG4J_PATTERN sets the logging pattern (default: [%d] (%t) %p %m (%c)%n)
• CONNECT_LOG4J_ROOT_LOGLEVEL sets the root log level (default: INFO)
• CONNECT_LOG4J_LOGGERS is a comma separated list of logger and log level key-value pairs (default: org.reflections=ERROR,org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR)

⁠JMX

Remote JMX can be enabled with the following environment variables:

KAFKA_JMX_PORT=6001
KAFKA_JMX_HOSTNAME=localhost

⁠Debugging

In order to debug Kafka Connect, set the following environment variable:

KAFKA_DEBUG=y

In addition you can configure the behavior with the following environment variables:

JAVA_DEBUG_PORT=5005
DEBUG_SUSPEND_FLAG=y

⁠Verbatim Connector Configuration

Some configurations cannot be converted to the environment variable key/value schema. This is the case for example, if camel-case has been used for configuration variables, e.g. transforms.expandvalue.sourceFields=value.

To support configurations like this, you can define environment variable with CONNECTORPROPERTIES_ as name prefix. Any content is added to the connector configuration as is.

The following shows an example for a SMT configuration.

CONNECTORPROPERTIES_TRANSFORMS: |
    transforms.expandvalue.type=com.redhat.insights.expandjsonsmt.ExpandJSON$$Value
    transforms.expandvalue.sourceFields=value

You can find the entire example setup at examples/connect-standalone/http-source-plugin-install/compose.yaml⁠.

⁠Kafka Connect Plugin Installation

The Apache Kafka Connect Container image supports installation of Kafka Connect plugins like connectors during startup with multiple methods.

Define a comma separated list of plugins which should be installed via Confluent Hub.

PLUGIN_INSTALL_CONFLUENT_HUB_IDS: |
    confluentinc/kafka-connect-jdbc:latest
    confluentinc/kafka-connect-http:latest

Define a comma separated list of plugin Urls. Supported are *.zip, *.tar*, *.tgz and *.jar files.

PLUGIN_INSTALL_URLS: |
    https://github.com/castorm/kafka-connect-http/releases/download/v0.8.11/castorm-kafka-connect-http-0.8.11.zip
    https://github.com/RedHatInsights/expandjsonsmt/releases/download/0.0.7/kafka-connect-smt-expandjsonsmt-0.0.7.tar.gz

Define a comma separated list of 'path=url' pairs, to download additional libraries. Supported are *.zip, *.tar*, *.tgz and *.jar files.

PLUGIN_INSTALL_LIB_URLS: |
    confluentinc-kafka-connect-jdbc/lib=https://dlm.mariadb.com/1496775/Connectors/java/connector-java-2.7.2/mariadb-java-client-2.7.2.jar
    confluentinc-kafka-connect-avro-converter/lib=https://repo1.maven.org/maven2/com/google/guava/guava/30.1.1-jre/guava-30.1.1-jre.jar

⁠Pre-installed Kafka Connect Plugins

This Kafka Connect image has the Confluent converters for Avro, Protobuf and JSON Schema already pre-installed to simplify usage of Confluent Schema Registry.

• confluentinc/kafka-connect-avro-converter:7.4.0⁠
• confluentinc/kafka-connect-protobuf-converter:7.4.0⁠
• confluentinc/kafka-connect-json-schema-converter:7.4.0⁠

This Kafka Connect image has the Confluent Connect SMTs already pre-installed:

• confluentinc/connect-transforms:1.4.3⁠

⁠Build

In order to create your own Container image for Apache Kafka Connect standalone clone the ueisele/kafka-image⁠ Git repository and run the build command:

git clone https://gitlab.com/ueisele/kafka-images.git
cd kafka-images
connect-standalone/build.sh --build --tag 3.5.0 --openjdk-release 20

To create an image with a specific OpenJDK version use the following command:

connect-standalone/build.sh --build --tag 3.5.0 --openjdk-release 20 --openjdk-version 20.0.1

To build the most recent SNAPSHOT of Apache Kafka 3.5.0 with Java 17, run:

connect-standalone/build.sh --build --branch trunk --openjdk-release 17

⁠Build Options

The connect-standalone/build.sh script provides the following options:

Usage: connect-standalone/build.sh [--build] [--push] [--registry docker.io] [--user ueisele] [--archs amd64,arm64] [--github-repo apache/kafka] [--commit-sha 09e8adb] [--tag 3.5.0] [--branch trunk] [--pull-request 9999] [--openjdk-release 20] [--openjdk-version 20.0.1]

⁠License

This Container image is licensed under the Apache 2 license⁠.