mmolimar/vkitm repository overview

⁠VKitM - Virtual Kafka in the Middle

VKitM is a Apache Kafka⁠ proxy which uses the native Kafka protocol.

Its name is due to how it has been designed: the application acts as an intermediary between the Kafka client and the Kafka cluster, modifying the requests and responses from/to the client. Actually, the client thinks that the connection is to a Kafka cluster (by now, with one node) and doesn't know what's behind (an entirely Kafka cluster).

Right now, the implementation includes a subset of the API Keys⁠ but can produce and consume messages.

You can find the source code at GitHub⁠.

⁠How it works

1. Kafka client connects to a single Kafka broker. Actually, it's connecting to a VKitM.
2. VKitM modifies the request, depending of the API Key.
3. The modified request is sent to the actual Kafka cluster, if applicable.
4. VKitM receives the response from the Kafka brokers.
5. Based on the kind of response it has to send to the client, VKitM changes data related with brokers, partition leaders and other metadata in the final response.
6. The client receives the response thinking that there is just one node in the cluster and continues.

DISCLAIMER: This application has been implemented in order to provide another option when a proxy is required. If you use VKitM with a different purpose from the one has been designed it's under your own responsibility.

⁠Motivation

There are multiple Kafka proxies out there but most of them are HTTP REST based. A well-known REST Proxy is this one⁠.

However, most of them lack of some of the following features I'd like to provide in VKitM:

• Use the native Kafka protocol directly, improving the performance.
• Be able to modify the messages from the clients on the fly when we cannot do it in the client itself (due to business cases or whatever) including interceptors⁠.
• Define custom ACLs to complement those that already exist (or maybe not) in the Kafka Cluster.
• Change the kind of protocol in your clients (secured or not). I.e.: using TLS connection from the clients to VKitM and from the VKitM to Kafka brokers in plaintext (or vice versa) or even change the certificates they use.

⁠Running VKitM Docker image

docker run -p <HOST_PORT>:<CONTAINER_PORT> -v /path/to/conf/directory:/vkitm/conf mmolimar/vkitm

NOTE: path /path/to/conf/directory must contain a file named application.conf with the VKitM configuration.

⁠Sample configuration file

server {
  listeners = "PLAINTEXT://0.0.0.0:19092"
  num.network.threads = 3
  num.io.threads = 8
  zookeeper.connect = "<HOST_ZK_IP>:<HOST_ZK_PORT>"
  zookeeper.connection.timeout.ms = 6000
}

producer {
  bootstrap.servers = "<HOST_KAFKA_IP>:<HOST_KAFKA_PORT>"
  compression.type = "none"
}

consumer {
  zookeeper.connect = "<HOST_ZK_IP>:<HOST_ZK_PORT>"
  zookeeper.connection.timeout.ms = 6000
  group.id = "consumer-group"
}