abrahamleal/kafka-idle-topics repository overview

= kafka-idle-topics

image:https://travis-ci.com/abraham-leal/kafka-idle-topics.svg?branch=main["Build⁠ Status", link="https://travis-ci.com/abraham-leal/kafka-idle-topics⁠"]

Tooling to find topics that are not utilized in your Kafka cluster.

Ensure efficient usage of your topics in https://www.confluent.io/confluent-cloud/[Confluent⁠ Cloud] by discovering topics that aren't being used.

kafka-idle-topics evaluates an idle topic through these criteria:

• It has no Consumer Groups (including inactive consumer groups that have offsets committed).
• It has no active production during the time of assessment (30 seconds while running by default).
• It has no data within its partitions.

To download, go to Run for different ways of usage. It is also possible to download a release. The output will be written to a file in the same directory. The name of this file is configurable.

== Usage

⁠[source,bash]

⁠Usage of ./kafka-idle-topics: -bootstrap-servers string Address to the target Kafka Cluster. Accepts multiple endpoints separated by a comma. -filename string Custom filename for the output if needed. (default "idleTopics.txt") -hideInternalTopics Hide internal topics from assessment. -idleMinutes int Amount of minutes a topic should be idle to report it. -kafkaSecurity string Type of connection to attempt. Options: plain_tls, plain (no tls), tls (one-way), none. (default "none") -password string Password in the PLAIN module. -productionAssessmentTimeMs int Timeframe to assess active production. (default 30000) -skip string Filtering to skip. Options are: production, consumption, storage. This can be a comma-delimited list. -username string Username in the PLAIN module.

== Security

kafka-idle-topics supports the following security set-ups:

• plain_tls: SASL_SSL w/ PLAIN
• plain: SASL_PLAINTEXT w/ PLAIN (No TLS)
• tls: SSL (One-way TLS)
• none: PLAINTEXT (Default)

To add certificates the tool can trust for TLS connections, ensure they are located in the subdirectories go seeks. Commonly it is /etc/ssl/certs/.

== Run

You can run kafka-idle-topics in multiple configurations:

=== Docker

Example with Confluent Cloud:

⁠[source,bash]

⁠docker run
-e KAFKA_BOOTSTRAP=<>
-e KAFKA_USERNAME=<>
-e KAFKA_PASSWORD=<>
abrahamleal/kafka-idle-topics:latest

=== Locally

Downloading with Go, example with Confluent Cloud:

⁠[source,bash]

⁠go get github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics go install github.com/abraham-leal/kafka-idle-topics/cmd/kafka-idle-topics $GOPATH/bin/kafka-idle-topics -bootstrap-servers <> -username <> -password <> -kafkaSecurity plain_tls

=== Build From Source

Git clone, example with Confluent Cloud:

⁠[source,bash]

⁠git clone https://github.com/abraham-leal/kafka-idle-topics⁠ cd kafka-idle-topics/ go build ./cmd/kafka-idle-topics/kafka-idle-topics.go ./kafka-idle-topics -bootstrap-servers <> -username <> -password <> -kafkaSecurity plain_tls

== Feature Requests / Issue Reporting

This repo tracks feature requests and issues through Github Issues. If you'd like to see something fixed that was not caught by testing, or you'd like to see a new feature, please feel free to file a Github issue in this repo, I'll review and answer at best effort.

Additionally, if you'd like to contribute a fix/feature, please feel free to open a PR for review.