n8500x/kafka-installer repository overview

⁠Kafka + ZooKeeper (Argo CD, air-gapped)

A 3-node Kafka cluster backed by a 3-node ZooKeeper ensemble, deployed as two separate Argo CD Applications. Charts are vendored into a Docker image for air-gapped install. Client and inter-broker traffic uses SASL_SSL + SCRAM-SHA-512 (SCRAM over TLS; JKS certs auto-generated by the installer with keytool). Sized to run on a single minikube VM.

⁠Why these versions (read this first)

Kafka 4.0 (Bitnami chart 28+/32+) removed ZooKeeper entirely and is KRaft-only — and KRaft does not support SCRAM. Because this cluster needs both external ZooKeeper and SCRAM-SHA-512, we deliberately pin Kafka chart 27.1.2 (Kafka 3.7), the last Bitnami line that supports kraft.enabled=false

• external ZK + SCRAM. This is an EOL Kafka line — fine for a pinned, air-gapped deployment, but a conscious choice.

⁠Layout

kafka/
  zookeeper/            # Argo CD app source #1 (umbrella chart)
    Chart.yaml          #   dependency: zookeeper 13.8.7
    values.yaml         #   3-node ensemble, minikube-sized
    charts/zookeeper-13.8.7.tgz     # vendored subchart (no network at sync)
  kafka/                # Argo CD app source #2 (umbrella chart)
    Chart.yaml          #   dependency: kafka 27.1.2
    values.yaml         #   3 brokers, ZK mode, SASL/SCRAM-SHA-512
    charts/kafka-27.1.2.tgz         # vendored subchart
  argocd/               # standalone Application manifests (direct apply)
    zookeeper-app.yaml
    kafka-app.yaml
  test/                 # integration harness (produce/consume + assert)
    integration-job.yaml
    run-integration.sh
  render.sh             # in-image helper: certs | template | install | argocd
  Dockerfile            # bakes both charts + test into n8500x/kafka-installer
  deploy-minikube.sh    # wire the two apps into Argo CD on minikube
  ONPREM.md             # exact air-gapped bring-up commands

Air-gapped on-prem bring-up is a separate, exact runbook: ONPREM.md⁠.

⁠Install in one command

Argo CD already running in the cluster? This emits the namespace, the JKS TLS secrets and both Applications as a single stream and hands it to your kubectl — no git clone, no helm dependency update, no local charts:

docker run --rm -e REPO_URL=https://github.com/you/docker.git -e REVISION=master \
  --entrypoint /kafka/render.sh n8500x/kafka-installer:latest argocd | kubectl apply -f -

REPO_URL must be a git remote Argo CD can reach that holds this repo (Argo CD pulls charts from git, not from the image); the apps track kafka/zookeeper and kafka/kafka in it. ZooKeeper is sync-wave 0, Kafka sync-wave 1, so the ensemble is up before the brokers register. Re-running rotates the demo TLS certs — add -e SKIP_CERTS=1 to keep the ones already in the cluster.

No Argo CD (or no git remote)? The same image installs both charts directly over a mounted kubeconfig, also in one command:

docker run --rm -v $HOME/.kube:/root/.kube:ro \
  --entrypoint /kafka/render.sh n8500x/kafka-installer:latest install

The two apps also appear in the shared app-of-apps (argocd/appofapps/values.yaml⁠) as zookeeper (sync-wave 0) and kafka (sync-wave 1), both targeting the kafka namespace.

⁠What's in the image, and what isn't

Everything needed to render and install is baked under /kafka:

/kafka/zookeeper/{Chart.yaml,values.yaml,charts/zookeeper-13.8.7.tgz}
/kafka/kafka/{Chart.yaml,values.yaml,charts/kafka-27.1.2.tgz}
/kafka/argocd/{zookeeper-app.yaml,kafka-app.yaml}
/kafka/test/{integration-job.yaml,run-integration.sh}
/kafka/render.sh

The subcharts are vendored, not referenced, so no helm dependency update and no registry round-trip happen at render time. Verify that yourself with no network at all:

docker run --rm --network none \
  --entrypoint /kafka/render.sh n8500x/kafka-installer:latest template

Two things are deliberately not in the image:

1. The runtime container images. The cluster pulls these itself; only two are referenced (provisioning and metrics are off, so no os-shell / kubectl images are needed):

Bitnami reorganised its catalog in 2025 and deleted the original tags from docker.io/bitnami, so these live under bitnamilegacy (which is why global.security.allowInsecureImages: true is set — the chart hard-fails rendering on an "unrecognised" repo otherwise). For a real air gap, mirror both into your registry and set global.imageRegistry; see ONPREM.md⁠.

2. The charts Argo CD actually renders. On the argocd path the image only emits the Application manifests — Argo CD's repo-server then clones REPO_URL and renders kafka/zookeeper and kafka/kafka from git, not from the image. The vendored .tgz files are committed to the repo too, so that side is equally offline, but it does mean the GitOps path needs a reachable git remote and the image's own copy of the charts goes unused. The install path is the one that renders from the baked charts.

⁠Build the installer image

docker build -t n8500x/kafka-installer:27.1.2 kafka
docker save n8500x/kafka-installer:27.1.2 -o kafka-installer-27.1.2.tar   # for transport

Inspect or install without Argo CD (certs mints the JKS TLS secrets):

# generate the JKS TLS secrets (keytool) -> kafka-tls + kafka-tls-passwords
docker run --rm --entrypoint /kafka/render.sh n8500x/kafka-installer:27.1.2 certs
# render both charts to stdout (references the kafka-tls secret)
docker run --rm --entrypoint /kafka/render.sh n8500x/kafka-installer:27.1.2 template
# create ns, generate+apply the TLS secrets (idempotent), then helm-install ZK then Kafka
docker run --rm -v $HOME/.kube:/root/.kube:ro \
  --entrypoint /kafka/render.sh n8500x/kafka-installer:27.1.2 install

⁠Deploy on minikube via Argo CD (see it in the UI)

Argo CD pulls charts from git, not local disk, so push this monorepo to a remote your cluster can reach and point the apps at it.

The one-command install above is the short path. deploy-minikube.sh is the scripted equivalent for a checkout — it also refreshes the minikube context and can install Argo CD itself:

minikube start --memory 8192 --cpus 4          # 6 stateful pods + Argo CD need headroom
# Argo CD must be installed in ns 'argocd' (use ../argocd, or INSTALL_ARGOCD=1 below)

REPO_URL=https://github.com/you/docker.git REVISION=master \
  ./kafka/deploy-minikube.sh
# add INSTALL_ARGOCD=1 to render+apply the vendored argo-cd chart first

Then watch it converge:

kubectl -n argocd get applications -w      # zookeeper -> Synced/Healthy, then kafka
kubectl -n kafka  get pods -w              # kafka-zookeeper-0..2, kafka-broker-0..2

Argo CD UI: port-forward svc/argocd-server (8080:443) and log in as admin with the initial-admin-secret password (both commands are printed by the script). If Avast blocks kubectl port-forward, SSH-tunnel to the ClusterIP through minikube's SSH port instead (repo memory: minikube_ui_access_ssh_tunnel).

⁠Connecting with SASL_SSL + SCRAM-SHA-512

Bootstrap server (in-cluster): kafka.kafka.svc.cluster.local:9092. Over SASL_SSL, clients need the JKS truststore (and its password) from the auto-generated kafka-tls secret:

kubectl -n kafka get secret kafka-tls -o jsonpath='{.data.kafka\.truststore\.jks}' | base64 -d > kafka.truststore.jks
TRUSTPW=$(kubectl -n kafka get secret kafka-tls-passwords -o jsonpath='{.data.truststore-password}' | base64 -d)

Demo credentials (defined in kafka/values.yaml⁠):

client.properties:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="user1" password="user1-demo-pass";
ssl.truststore.type=JKS
ssl.truststore.location=/tmp/kafka.truststore.jks
ssl.truststore.password=<truststore-password>
ssl.endpoint.identification.algorithm=

ssl.endpoint.identification.algorithm= (empty) disables hostname verification — the certs are self-signed and reached via localhost / service DNS.

Smoke test from a broker pod:

kubectl -n kafka cp kafka.truststore.jks kafka-broker-0:/tmp/kafka.truststore.jks
kubectl -n kafka cp client.properties     kafka-broker-0:/tmp/client.properties
kubectl -n kafka exec -it kafka-broker-0 -- \
  kafka-topics.sh --bootstrap-server localhost:9092 --command-config /tmp/client.properties \
  --create --topic demo --partitions 3 --replication-factor 3

⁠Refreshing the vendored charts

helm pull oci://registry-1.docker.io/bitnamicharts/zookeeper --version 13.8.7 \
  -d kafka/zookeeper/charts
helm pull oci://registry-1.docker.io/bitnamicharts/kafka --version 27.1.2 \
  -d kafka/kafka/charts

Keep the version: in each Chart.yaml in sync with the vendored .tgz.