A lightweight high-level kafka replicator based on kafka-node
254
This software is very new, and not ready for production use.
Replikafka is a lightweight kafka replicator build on top of kafka-node.
By replication we mean to continiously copy messages from (a list of topics on) a source kafka cluster to (a list of the same or different topics on) a destination Kafka cluster.
Isn't that what MirrorMaker is for?
Yes, Kafka MirrorMaker is a well-established solution that does the same thing. But there's a catch.
MirrorMaker requires a connection to Zookeeper to mirror topics. Since Kafka version 0.9, we have the concept of high-level consumers/producers. With these, a Zookeeper connection is no longer required for replication/mirroring.
However, MirrorMaker predates this, and while there is a proposal for MirrorMaker 2, it has yet to be build.
MirrorMaker's shortcomings have spawned a number of alternatives that are worth looking in to:
- Uber's uReplicator
- Confluent's Replicator
- ComCast's Mirror tool for Kafka Connect
- Salesforce's Mirus
They are typically build by large companies that need to mirror some trillions of messages. As a result, setting them up can be a daunting process.
I needed something for a smaller-scale deployment that didn't require a connection to Zookeeper. From the start, I wanted to be able to just spin up a container, pass in a config file, and not look back.
Configuration is handled via a single YAML file.
By default, it is loaded from /etc/replikafka/config.yml but you can change that in src/config.js.
If you run this in a container, mount the config file as a volume.
Example:
# Source: the Kafka cluster you read messages from
source:
# The consumer-group name. Allows running multiple instances
# for horizontal scaling. Just give them the same name to
# distribute the load between them.
groupId: replikafka
# The list of kafka brokers for your source cluster
brokers:
- 1.kafka.changeme.com
- 2.kafka.changeme.com
- 3.kafka.changeme.com
# Port of the brokers
port: 9093
# Topics to replicate
topics:
- in
- test
# Timeouts for connections and requests. Remove entirely
# to use the kafka-node defaults
timeouts:
connect: 5000
request: 5000
# SSL settings. Remove entirely to use plain-text connections
ssl:
# Set to true (or remove) to enforce certificate validation
# Set to false to accept certificate regardless of validation,
# which is handy for testing with a self-signed certificate
verifyCert: false
# If you're using client certificates for authentication
# configure the path to the client certificate and key here
clientAuth:
key: /etc/replikafka/client.key
cert: /etc/replikafka/client.crt
# The key's passphrase should not be included in
# the configuration. Instead point to a file that contains
# the passphrase. This way, you can keep your config under
# version control
passphrase: /etc/replikafka/passphrase
# If your brokers use a certificate that's not trusted,
# you can specify a file with CA certificates to trust.
# Note that this will overwrite the default trusted CAs
ca: /etc/replikafka/ca-list.pem
# Destination: the Kafka cluster you send messages to
# The configuration is the same as the source cluster, except:
# - There is no groupId as that makes no sense here
# - The topics configuration determines how we replicate
destination:
brokers:
- 1.kafka.changeme.com
- 2.kafka.changeme.com
- 3.kafka.changeme.com
port: 9093
# If you remove topics entirely, messages will be replicated
# from source topic to the same destination topic
# If topics is configured, it should be key-value pairs
# that determine where to re-route messages.
# For example, in the configation below, the source 'in' topic
# will end up at the destination 'out' topic
# The source 'test' topic will end up at the destination 'test' topic
# because there is no re-routing for it.
# In other words, you only have to configure what you want to be
# re-routed. Anything else will end up in the topic with the same name
# as the source topic
topics:
in: out
timeouts:
connect: 5000
request: 5000
ssl:
verifyCert: false
clientAuth:
key: /etc/replikafka/client.key
certificate: /etc/replikafka/client.crt
passphrase: /etc/replikafka/passphrase
ca: /etc/replikafka/ca-list.pem
Enable debugging by setting the DEBUG environment variable:
export DEBUG=kafka-node:*
Or, if you are using a container, pass it to the container
via -e "DEBUG=kafka-node:*".
If you run into trouble, create and issue.
© Joost De Cock.
See the license file for details.
Content type
Image
Digest
Size
52.6 MB
Last updated
almost 7 years ago
docker pull joostdecock/replikafka