bitnamicharts/kafka
Bitnami Helm chart for Apache Kafka
5M+
Apache Kafka is a distributed streaming platform designed to build real-time pipelines and can be used as a message broker or as a replacement for a log aggregation solution for big data applications.
Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.
helm install my-release oci://registry-1.docker.io/bitnamicharts/kafka
Looking to use Apache Kafka in production? Try VMware Tanzu Application Catalog, the commercial edition of the Bitnami catalog.
This chart bootstraps a Kafka deployment on a Kubernetes cluster using the Helm package manager.
Bitnami charts can be used with Kubeapps for deployment and management of Helm Charts in clusters.
To install the chart with the release name my-release
:
helm install my-release oci://REGISTRY_NAME/REPOSITORY_NAME/kafka
Note: You need to substitute the placeholders
REGISTRY_NAME
andREPOSITORY_NAME
with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to useREGISTRY_NAME=registry-1.docker.io
andREPOSITORY_NAME=bitnamicharts
.
These commands deploy Kafka on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.
Tip: List all releases using
helm list
Bitnami charts allow setting resource requests and limits for all containers inside the chart deployment. These are inside the resources
value (check parameter table). Setting requests is essential for production workloads and these should be adapted to your specific use case.
To make this process easier, the chart contains the resourcesPreset
values, which automatically sets the resources
section according to different presets. Check these presets in the bitnami/common chart. However, in production workloads using resourcesPreset
is discouraged as it may not fully adapt to your specific needs. Find more information on container resource management in the official Kubernetes documentation.
This chart can be integrated with Prometheus by setting metrics.jmx.enabled
to true
. This will deploy a sidecar container with jmx_exporter in all pods and a metrics
service, which can be configured under the metrics.service
section. This metrics
service will have the necessary annotations to be automatically scraped by Prometheus.
Prometheus requirements
It is necessary to have a working installation of Prometheus or Prometheus Operator for the integration to work. Install the Bitnami Prometheus helm chart or the Bitnami Kube Prometheus helm chart to easily have a working Prometheus in your cluster.
Integration with Prometheus Operator
The chart can deploy ServiceMonitor
objects for integration with Prometheus Operator installations. To do so, set the value metrics.serviceMonitor.enabled=true
. Ensure that the Prometheus Operator CustomResourceDefinitions
are installed in the cluster or it will fail with the following error:
no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
Install the Bitnami Kube Prometheus helm chart for having the necessary CRDs and the Prometheus Operator.
It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.
Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.
This chart allows you to automatically configure Kafka with 3 listeners:
For more complex configurations, set the listeners
, advertisedListeners
and listenerSecurityProtocolMap
parameters as needed.
You can configure different authentication protocols for each listener you configure in Kafka. For instance, you can use sasl_tls
authentication for client communications, while using tls
for inter-broker communications. This table shows the available protocols and the security they provide:
Method | Authentication | Encryption via TLS |
---|---|---|
plaintext | None | No |
tls | None | Yes |
mtls | Yes (two-way authentication) | Yes |
sasl | Yes (via SASL) | No |
sasl_tls | Yes (via SASL) | Yes |
Configure the authentication protocols for client and inter-broker communications by setting the auth.clientProtocol and auth.interBrokerProtocol parameters to the desired ones, respectively.
If you enabled SASL authentication on any listener, you can set the SASL credentials using the parameters below:
auth.sasl.jaas.clientUsers
/auth.sasl.jaas.clientPasswords
: when enabling SASL authentication for communications with clients.auth.sasl.jaas.interBrokerUser
/auth.sasl.jaas.interBrokerPassword
: when enabling SASL authentication for inter-broker communications.auth.jaas.zookeeperUser
/auth.jaas.zookeeperPassword
: In the case that the Zookeeper chart is deployed with SASL authentication enabled.In order to configure TLS authentication/encryption, you can create a secret per Kafka broker you have in the cluster containing the Java Key Stores (JKS) files: the truststore (kafka.truststore.jks
) and the keystore (kafka.keystore.jks
). Then, you need pass the secret names with the tls.existingSecret
parameter when deploying the chart.
Note: If the JKS files are password protected (recommended), you will need to provide the password to get access to the keystores. To do so, use the
tls.password
parameter to provide your password.
For instance, to configure TLS authentication on a Kafka cluster with 2 Kafka brokers use the commands below to create the secrets:
kubectl create secret generic kafka-jks-0 --from-file=kafka.truststore.jks=./kafka.truststore.jks --from-file=kafka.keystore.jks=./kafka-0.keystore.jks
kubectl create secret generic kafka-jks-1 --from-file=kafka.truststore.jks=./kafka.truststore.jks --from-file=kafka.keystore.jks=./kafka-1.keystore.jks
Note: the command above assumes you already created the truststore and keystores files. This script can help you with the JKS files generation.
If, for some reason (like using Cert-Manager) you can not use the default JKS secret scheme, you can use the additional parameters:
tls.jksTruststoreSecret
to define additional secret, where the kafka.truststore.jks
is being kept. The truststore password must be the same as in tls.password
tls.jksTruststore
to overwrite the default value of the truststore key (kafka.truststore.jks
).Note: If you are using cert-manager, particularly when an ACME issuer is used, the
ca.crt
field is not put in theSecret
that cert-manager creates. To handle this, thetls.pemChainIncluded
property can be set totrue
and the initContainer created by this Chart will attempt to extract the intermediate certs from thetls.crt
field of the secret (which is a PEM chain) Note: The truststore/keystore from above must be protected with the same password as intls.password
You can deploy the chart with authentication using the following parameters:
replicaCount=2
listeners.client.client.protocol=SASL
listeners.client.interbroker.protocol=TLS
tls.existingSecret=kafka-jks
tls.password=jksPassword
sasl.client.users[0]=brokerUser
sasl.client.passwords[0]=brokerPassword
sasl.zookeeper.user=zookeeperUser
sasl.zookeeper.password=zookeeperPassword
zookeeper.auth.enabled=true
zookeeper.auth.serverUsers=zookeeperUser
zookeeper.auth.serverPasswords=zookeeperPassword
zookeeper.auth.clientUser=zookeeperUser
zookeeper.auth.clientPassword=zookeeperPassword
You can deploy the chart with AclAuthorizer using the following parameters:
replicaCount=2
listeners.client.protocol=SASL
listeners.interbroker.protocol=SASL_TLS
tls.existingSecret=kafka-jks-0
tls.password=jksPassword
sasl.client.users[0]=brokerUser
sasl.client.passwords[0]=brokerPassword
sasl.zookeeper.user=zookeeperUser
sasl.zookeeper.password=zookeeperPassword
zookeeper.auth.enabled=true
zookeeper.auth.serverUsers=zookeeperUser
zookeeper.auth.serverPasswords=zookeeperPassword
zookeeper.auth.clientUser=zookeeperUser
zookeeper.auth.clientPassword=zookeeperPassword
authorizerClassName=kafka.security.authorizer.AclAuthorizer
allowEveryoneIfNoAclFound=false
superUsers=User:admin
If you are using Kafka ACLs, you might encounter in kafka-authorizer.log the following event: [...] Principal = User:ANONYMOUS is Allowed Operation [...]
.
By setting the following parameter: listeners.client.protocol=SSL
and listener.client.sslClientAuth=required
, Kafka will require the clients to authenticate to Kafka brokers via certificate.
As result, we will be able to see in kafka-authorizer.log the events specific Subject: [...] Principal = User:CN=kafka,OU=...,O=...,L=...,C=..,ST=... is [...]
.
The Bitnami Kafka chart, when upgrading, reuses the secret previously rendered by the chart or the one specified in sasl.existingSecret
. To update credentials, use one of the following:
helm upgrade
specifying new credentials in the sasl
section as explained in the authentication section.helm upgrade
specifying a new secret in sasl.existingSecret
In order to access Kafka Brokers from outside the cluster, an additional listener and advertised listener must be configured. Additionally, a specific service per kafka pod will be created.
There are three ways of configuring external access. Using LoadBalancer services, using NodePort services or using ClusterIP services.
Using LoadBalancer services
You have two alternatives to use LoadBalancer services:
externalAccess.enabled=true
externalAccess.broker.service.type=LoadBalancer
externalAccess.controller.service.type=LoadBalancer
externalAccess.broker.service.ports.external=9094
externalAccess.controller.service.containerPorts.external=9094
externalAccess.autoDiscovery.enabled=true
serviceAccount.create=true
rbac.create=true
Note: This option requires creating RBAC rules on clusters where RBAC policies are enabled.
externalAccess.enabled=true
externalAccess.controller.service.type=LoadBalancer
externalAccess.controller.service.containerPorts.external=9094
externalAccess.controller.service.loadBalancerIPs[0]='external-ip-1'
externalAccess.controller.service.loadBalancerIPs[1]='external-ip-2'
externalAccess.broker.service.type=LoadBalancer
externalAccess.broker.service.ports.external=9094
externalAccess.broker.service.loadBalancerIPs[0]='external-ip-3'
externalAccess.broker.service.loadBalancerIPs[1]='external-ip-4'
Note: You need to know in advance the load balancer IPs so each Kafka broker advertised listener is configured with it.
Following the aforementioned steps will also allow to connect the brokers from the outside using the cluster's default service (when service.type
is LoadBalancer
or NodePort
). Use the property service.externalPort
to specify the port used for external connections.
Using NodePort services
You have two alternatives to use NodePort services:
Option A) Use random node ports using an initContainer that discover them automatically.
externalAccess.enabled=true
externalAccess.controller.service.type=NodePort
externalAccess.broker.service.type=NodePort
externalAccess.autoDiscovery.enabled=true
serviceAccount.create=true
rbac.create=true
Note: This option requires creating RBAC rules on clusters where RBAC policies are enabled.
Option B) Manually specify the node ports:
externalAccess.enabled=true
externalAccess.controller.service.type=NodePort
externalAccess.controller.service.nodePorts[0]='node-port-1'
externalAccess.controller.service.nodePorts[1]='node-port-2'
Note: You need to know in advance the node ports that will be exposed so each Kafka broker advertised listener is configured with it.
The pod will try to get the external ip of the node using curl -s https://ipinfo.io/ip
unless externalAccess.<controller|broker>.service.domain
or externalAccess.<controller|broker>.service.useHostIPs
is provided.
Option C) Manually specify distinct external IPs (using controller+broker nodes)
externalAccess.enabled=true
externalAccess.controller.service.type=NodePort
externalAccess.controller.service.externalIPs[0]='172.16.0.20'
externalAccess.controller.service.externalIPs[1]='172.16.0.21'
externalAccess.controller.service.externalIPs[2]='172.16.0.22'
Note: You need to know in advance the available IP of your cluster that will be exposed so each Kafka broker advertised listener is configured with it.
Using ClusterIP services
Note: This option requires that an ingress is deployed within your cluster
externalAccess.enabled=true
externalAccess.controller.service.type=ClusterIP
externalAccess.controller.service.ports.external=9094
externalAccess.controller.service.domain='ingress-ip'
externalAccess.broker.service.type=ClusterIP
externalAccess.broker.service.ports.external=9094
externalAccess.broker.service.domain='ingress-ip'
Note: the deployed ingress must contain the following block:
tcp:
9094: "{{ include "common.names.namespace" . }}/{{ include "common.names.fullname" . }}-0-external:9094"
9095: "{{ include "common.names.namespace" . }}/{{ include "common.names.fullname" . }}-1-external:9094"
9096: "{{ include "common.names.namespace" . }}/{{ include "common.names.fullname" . }}-2-external:9094"
Name resolution with External-DNS
You can use the following values to generate External-DNS annotations which automatically creates DNS records for each ReplicaSet pod:
externalAccess:
controller:
service:
annotations:
external-dns.alpha.kubernetes.io/hostname: "{{ .targetPod }}.example.com"
The chart can optionally start two metrics exporters:
To expose JMX metrics to Prometheus, use the parameter below:
metrics.jmx.enabled: true
zookeeper.metrics.enabled: true
If you have a need for additional containers to run within the same pod as Kafka (e.g. an additional metrics or logging exporter), you can do so via the sidecars
config parameter. Simply define your container according to the Kubernetes container spec.
sidecars:
- name: your-image-name
image: your-image
imagePullPolicy: Always
ports:
- name: portname
containerPort: 1234
This chart allows you to set your custom affinity using the affinity
parameter. Find more information about Pod's affinity in the kubernetes documentation.
As an alternative, you can use of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common chart. To do so, set the podAffinityPreset
, podAntiAffinityPreset
, or nodeAffinityPreset
parameters.
There are cases where you may want to deploy extra objects, such as Kafka Connect. For covering this case, the chart allows adding the full specification of other objects using the extraDeploy
parameter. The following example would create a deployment including a Kafka Connect deployment so you can connect Kafka with MongoDB®:
## Extra objects to deploy (value evaluated as a template)
##
extraDeploy:
- |
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "common.names.fullname" . }}-connect
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
app.kubernetes.io/component: connector
spec:
replicas: 1
selector:
matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 6 }}
app.kubernetes.io/component: connector
template:
metadata:
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 8 }}
app.kubernetes.io/component: connector
spec:
containers:
- name: connect
image: KAFKA-CONNECT-IMAGE
imagePullPolicy: IfNotPresent
ports:
- name: connector
containerPort: 8083
volumeMounts:
- name: configuration
mountPath: /bitnami/kafka/config
volumes:
- name: configuration
configMap:
name: {{ include "common.names.fullname" . }}-connect
- |
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "common.names.fullname" . }}-connect
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
app.kubernetes.io/component: connector
data:
connect-standalone.properties: |-
bootstrap.servers = {{ include "common.names.fullname" . }}-0.{{ include "common.names.fullname" . }}-headless.{{ include "common.names.namespace" . }}.svc.{{ .Values.clusterDomain }}:{{ .Values.service.port }}
...
mongodb.properties: |-
connection.uri=mongodb://root:password@mongodb-hostname:27017
...
- |
apiVersion: v1
kind: Service
metadata:
name: {{ include "common.names.fullname" . }}-connect
labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
app.kubernetes.io/component: connector
spec:
ports:
- protocol: TCP
port: 8083
targetPort: connector
selector: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
app.kubernetes.io/component: connector
You can create the Kafka Connect image using the Dockerfile below:
FROM bitnami/kafka:latest
# Download MongoDB® Connector for Apache Kafka https://www.confluent.io/hub/mongodb/kafka-connect-mongodb
RUN mkdir -p /opt/bitnami/kafka/plugins && \
cd /opt/bitnami/kafka/plugins && \
curl --remote-name --location --silent https://search.maven.org/remotecontent?filepath=org/mongodb/kafka/mongo-kafka-connect/1.2.0/mongo-kafka-connect-1.2.0-all.jar
CMD /opt/bitnami/kafka/bin/connect-standalone.sh /opt/bitnami/kafka/config/connect-standalone.properties /opt/bitnami/kafka/config/mongo.properties
To back up and restore Helm chart deployments on Kubernetes, you need to back up the persistent volumes from the source deployment and attach them to a new deployment using Velero, a Kubernetes backup/restore tool. Find the instructions for using Velero in this guide.
The Bitnami Kafka image stores the Kafka data at the /bitnami/kafka
path of the container. Persistent Volume Claims are used to keep the data across deployments. This is known to work in GCE, AWS, and minikube.
As the image run as non-root by default, it is necessary to adjust the ownership of the persistent volume so that the container can write data into it.
By default, the chart is configured to use Kubernetes Security Context to automatically change the ownership of the volume. However, this feature does not work in all Kubernetes distributions. As an alternative, this chart supports using an initContainer to change the ownership of the volume before mounting it in the final destination.
You can enable this initContainer by setting volumePermissions.enabled
to true
.
Name | Description | Value |
---|---|---|
global.imageRegistry | Global Docker image registry | "" |
global.imagePullSecrets | Global Docker registry secret names as an array | [] |
global.defaultStorageClass | Global default StorageClass for Persistent Volume(s) |
Note: the README for this chart is longer than the DockerHub length limit of 25000, so it has been trimmed. The full README can be found at https://github.com/bitnami/charts/blob/main/bitnami/kafka/README.md