Klag is a service for monitoring Kafka consumer behaviour and lag. github.com/themoah/klag
8.2K
Know when your consumers fall behind, before it becomes a problem.
Klag continuously monitors all Kafka consumer groups and exports lag metrics to Prometheus, Datadog, or any OpenTelemetry-compatible backend. Inspired by kafka-lag-exporter (archived 2024). Built on Vert.x and Micrometer.
Scales to large clusters: monitors thousands of consumer groups in ~50MB heap. Configurable request batching prevents overwhelming brokers when monitoring 500+ groups.
docker run -e KAFKA_BOOTSTRAP_SERVERS=kafka:9092 \
-e METRICS_REPORTER=prometheus \
-p 8888:8888 \
themoah/klag:latest
Metrics available at http://localhost:8888/metrics
Consumer lag is the gap between what Kafka has produced and what your consumers have processed. Left unmonitored, growing lag leads to:
| Feature | Why It Matters |
|---|---|
| Lag velocity | Know if lag is growing or shrinking — catch problems before they escalate |
| Time-based lag estimation | See lag in seconds/minutes, not just message counts |
| Hot partition detection | Find partitions with uneven load causing bottlenecks |
| Data loss prevention | Alert before consumers fall past Kafka's retention window |
| Consumer group state tracking | Alert on Rebalancing, Dead, or Empty states |
| Request batching | Safely monitor large clusters without overwhelming brokers |
| Stale group cleanup | Automatically stops reporting deleted/inactive groups |
| Backend | METRICS_REPORTER value |
|---|---|
| Prometheus (scrape endpoint) | prometheus |
| Datadog | datadog |
| OTLP / OpenTelemetry (Grafana Cloud, New Relic, etc.) | otlp |
| Metric | Description |
|---|---|
klag.consumer.lag | Current lag per partition (also .sum, .max, .min) |
klag.consumer.lag.velocity | Rate of change — positive means falling behind |
klag.consumer.lag.ms | Lag converted to milliseconds using offset/timestamp interpolation |
klag.consumer.lag.time_to_close_seconds | Estimated seconds until lag reaches zero (catching-up groups only) |
klag.consumer.lag.retention_percent | Lag as % of available messages — use to prevent data loss |
klag.consumer.group.state | Group health: Stable, Rebalancing, Dead, Empty |
klag.hot_partition | Throughput × 100 for statistically abnormal partitions |
klag.hot_partition.lag | Lag on hot partitions specifically |
klag.topic.partitions | Partition count per topic |
klag.partition.log_end_offset | Latest offset per partition |
klag.consumer.committed_offset | Last committed offset per consumer |
All metrics are tagged with consumer_group, topic, and partition where applicable.
| Variable | Default | Description |
|---|---|---|
KAFKA_BOOTSTRAP_SERVERS | localhost:9092 | Kafka broker addresses |
KAFKA_SECURITY_PROTOCOL | (empty) | PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL |
KAFKA_SASL_MECHANISM | (empty) | PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 |
KAFKA_SASL_JAAS_CONFIG | (empty) | JAAS config string for SASL auth |
KAFKA_CHUNK_COUNT | 1 | Split offset requests into N batches (large clusters) |
KAFKA_CHUNK_DELAY_MS | 0 | Delay (ms) between batches |
METRICS_REPORTER | none | prometheus, datadog, or otlp |
METRICS_INTERVAL_MS | 60000 | Metric collection interval in milliseconds |
METRICS_GROUP_FILTER | * | Glob pattern to filter consumer groups |
METRICS_JVM_ENABLED | false | Expose JVM metrics (memory, GC, threads, CPU) |
HTTP_PORT | 8888 | HTTP server port |
METRICS_REPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64-encoded-credentials>
OTEL_SERVICE_NAME=klag-production
METRICS_REPORTER=datadog
DD_API_KEY=<your-api-key>
DD_SITE=datadoghq.com
docker run --env-file .env -p 8888:8888 themoah/klag:latest
Sample .env:
KAFKA_BOOTSTRAP_SERVERS=instance.gcp.confluent.cloud:9092
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=PLAIN
KAFKA_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=user password=pass;"
METRICS_REPORTER=prometheus
METRICS_INTERVAL_MS=30000
helm install klag ./charts/klag \
--set kafka.bootstrapServers="kafka-broker:9092"
The Helm chart includes liveness/readiness probes, ServiceMonitor support for Prometheus Operator, and SASL secret management.
| Endpoint | Purpose |
|---|---|
/healthz | Liveness probe — always 200 |
/readyz | Readiness probe — 200 if Kafka connected, 503 if not |
/metrics | Prometheus scrape endpoint (when METRICS_REPORTER=prometheus) |
Klag requires read-only Kafka access. It uses only the Kafka Admin Client API with DESCRIBE permissions — no write or alter access needed.
Required ACLs:
| Resource | Permission |
|---|---|
CLUSTER | DESCRIBE — health check, list consumer groups |
TOPIC * | DESCRIBE — partition info and offsets |
GROUP * | DESCRIBE — group state and committed offsets |
Works with self-managed Kafka, Confluent Cloud, and MSK.
A pre-built Grafana dashboard (dashboard/demo-dashboard.json) is included in the GitHub repository with 40 panels covering:
Import via Grafana → Dashboards → Import → Upload JSON.
linux/amd64, linux/arm648888-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0ghcr.io/themoah/klag:latestLicensed under the Apache 2.0 License.
Content type
Image
Digest
sha256:610579160…
Size
127 MB
Last updated
10 days ago
docker pull themoah/klag