n8500x/kexporter

By n8500x

•Updated 7 months ago

Kafka Connect metrics exporter (Prometheus) - runs as a sidecar next to Kafka Connect

Image

453

Overview Tags

n8500x/kexporter repository overview

⁠Kafka Connect Metrics Exporter

A Prometheus metrics exporter for Apache Kafka Connect, exposing connector and task metrics.

⁠Features

Collects metrics from Kafka Connect REST API
Exposes metrics in Prometheus format
Tracks connector and task states
Health check endpoint
Configurable scrape interval

⁠Metrics

⁠Exposed Metrics

kconnect_connectors_total - Total number of connectors
kconnect_connector_status - Connector status (1=running, 0=failed, -1=paused)
kconnect_tasks_total - Total tasks per connector
kconnect_task_status - Task status (1=running, 0=failed)
kconnect_collection_errors_total - Total collection errors
kconnect_collection_duration_seconds - Metric collection duration

⁠Configuration

Environment variables (all overridable):

KAFKA_CONNECT_URL - Kafka Connect REST API URL (default: http://localhost:8083)
METRICS_PORT - Port to expose Prometheus metrics (default: 8080)
SCRAPE_INTERVAL - Interval between metric collections in seconds (default: 30)
KAFKA_CONNECT_USER - Username for basic authentication (optional)
KAFKA_CONNECT_PASSWORD - Password for basic authentication (optional)

⁠Override Examples

Docker CLI (Basic Authentication):

docker run -d \
  --name kexporter \
  -p 9090:9090 \
  -e KAFKA_CONNECT_URL=http://kafka-connect:8083 \
  -e KAFKA_CONNECT_USER=admin \
  -e KAFKA_CONNECT_PASSWORD=secret123 \
  -e METRICS_PORT=9090 \
  -e SCRAPE_INTERVAL=60 \
  kexporter:v1.0

Docker Compose:

services:
  kexporter:
    image: kexporter:v1.0
    ports:
      - "8080:8080"
    environment:
      KAFKA_CONNECT_URL: http://kafka-connect:8083
      KAFKA_CONNECT_USER: admin
      KAFKA_CONNECT_PASSWORD: secret123
      METRICS_PORT: 8080
      SCRAPE_INTERVAL: 30

Kubernetes (Helm):

helm install kexporter ./helm/kexporter \
  --set kafkaConnect.url=http://kafka-connect:8083 \
  --set kafkaConnect.auth.enabled=true \
  --set kafkaConnect.auth.username=admin \
  --set kafkaConnect.auth.password=secret123 \
  --set exporter.metricsPort=8080 \
  --set exporter.scrapeInterval=30

⁠Usage

⁠Docker Build

docker build -t kexporter:latest .

⁠Docker Run

docker run -d \
  --name kexporter \
  -p 8080:8080 \
  -e KAFKA_CONNECT_URL=http://kafka-connect:8083 \
  -e METRICS_PORT=8080 \
  -e SCRAPE_INTERVAL=30 \
  kexporter:latest

⁠Kubernetes with Helm

⁠Quick Start

# Install with defaults
helm install kexporter ./helm/kexporter

# Install with custom values
helm install kexporter ./helm/kexporter \
  --set kafkaConnect.url=http://kafka-connect:8083 \
  --set exporter.metricsPort=8080 \
  --set exporter.scrapeInterval=30

# Install with basic authentication (secret auto-created)
helm install kexporter ./helm/kexporter \
  --set kafkaConnect.url=http://kafka-connect:8083 \
  --set kafkaConnect.auth.enabled=true \
  --set kafkaConnect.auth.username=admin \
  --set kafkaConnect.auth.secret.password=mysecretpassword

# Install with existing secret
kubectl create secret generic kexporter-auth --from-literal=password=mysecretpassword
helm install kexporter ./helm/kexporter \
  --set kafkaConnect.url=http://kafka-connect:8083 \
  --set kafkaConnect.auth.enabled=true \
  --set kafkaConnect.auth.username=admin \
  --set kafkaConnect.auth.secret.existingSecret=kexporter-auth

# Install with custom namespace
helm install kexporter ./helm/kexporter -n kafka-monitoring --create-namespace

# Upgrade existing release
helm upgrade kexporter ./helm/kexporter --set kafkaConnect.url=http://new-url:8083

# Uninstall
helm uninstall kexporter

⁠Values Customization

Create custom-values.yaml:

replicaCount: 3

kafkaConnect:
  url: http://kafka-connect-prod:8083
  timeout: 15
  auth:
    enabled: true
    username: admin
    secret:
      # Option 1: Auto-create secret with password
      password: mysecretpassword
      # Option 2: Use existing secret
      # existingSecret: kexporter-auth
      secretKey: password

exporter:
  metricsPort: 9090
  scrapeInterval: 60

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

# Enable Prometheus ServiceMonitor integration
serviceMonitor:
  enabled: true
  interval: 30s
  labels:
    release: prometheus

# Enable ingress
ingress:
  enabled: true
  className: nginx
  hosts:
    - host: kexporter.example.com
      paths:
        - path: /
          pathType: Prefix

Install with custom values:

helm install kexporter ./helm/kexporter -f custom-values.yaml

⁠Helm Values Reference

See helm/kexporter/values.yaml for all available configuration options.

⁠Kubernetes (sidecar)

Most deployments run kexporter as a sidecar in the same pod as Kafka Connect: it reaches the Connect REST API over localhost:8083 (the default KAFKA_CONNECT_URL) and serves metrics on :8080, which Prometheus scrapes straight from the pod.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-connect
spec:
  selector:
    matchLabels: { app: kafka-connect }
  template:
    metadata:
      labels: { app: kafka-connect }
      annotations:                       # let Prometheus scrape the exporter port
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: connect
          image: confluentinc/cp-kafka-connect:7.9.0
          ports:
            - { containerPort: 8083 }
          # ... your Kafka Connect config ...
        - name: kexporter               # the sidecar
          image: n8500x/kexporter:latest
          env:
            - { name: KAFKA_CONNECT_URL, value: "http://localhost:8083" }
            - { name: METRICS_PORT,      value: "8080" }
            - { name: SCRAPE_INTERVAL,   value: "30" }
          ports:
            - { name: metrics, containerPort: 8080 }
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 200m, memory: 128Mi }

For basic-auth-protected Connect, add KAFKA_CONNECT_USER and a KAFKA_CONNECT_PASSWORD sourced from a Secret via valueFrom.secretKeyRef. With the Prometheus Operator, drop the pod annotations and add a ServiceMonitor targeting the metrics port instead (the Helm chart's serviceMonitor.enabled=true does this for the standalone Deployment).

⁠Prometheus Configuration

Add to prometheus.yml:

scrape_configs:
  - job_name: 'kafka-connect'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/metrics'
    scrape_interval: 30s

⁠Endpoints

GET /metrics - Prometheus metrics
GET /health - Health check (returns JSON status)

⁠Requirements

Python 3.10+
Kafka Connect REST API accessible
Network connectivity to Kafka Connect cluster

⁠Example Prometheus Queries

# Connector count
kconnect_connectors_total

# Running connectors
count(kconnect_connector_status{connector_status="1"})

# Failed tasks
count(kconnect_task_status{status="0"})

# Collection latency
kconnect_collection_duration_seconds

⁠License

MIT

Tag summary

Recent tags

Content type

Image

Digest

sha256:f674d9f2c…

Size

52.1 MB

Last updated

7 months ago

docker pull n8500x/kexporter