bitnamicharts/etcd
Bitnami Helm chart for etcd
1M+
etcd is a distributed key-value store designed to securely store data across a cluster. etcd is widely used in production on account of its reliability, fault-tolerance and ease of use.
Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.
helm install my-release oci://registry-1.docker.io/bitnamicharts/etcd
Looking to use Etcd in production? Try VMware Tanzu Application Catalog, the commercial edition of the Bitnami catalog.
This chart bootstraps a etcd deployment on a Kubernetes cluster using the Helm package manager.
Bitnami charts can be used with Kubeapps for deployment and management of Helm Charts in clusters.
To install the chart with the release name my-release
:
helm install my-release oci://REGISTRY_NAME/REPOSITORY_NAME/etcd
Note: You need to substitute the placeholders
REGISTRY_NAME
andREPOSITORY_NAME
with a reference to your Helm chart registry and repository. For example, in the case of Bitnami, you need to useREGISTRY_NAME=registry-1.docker.io
andREPOSITORY_NAME=bitnamicharts
.
These commands deploy etcd on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.
Tip: List all releases using
helm list
Bitnami charts allow setting resource requests and limits for all containers inside the chart deployment. These are inside the resources
value (check parameter table). Setting requests is essential for production workloads and these should be adapted to your specific use case.
To make this process easier, the chart contains the resourcesPreset
values, which automatically sets the resources
section according to different presets. Check these presets in the bitnami/common chart. However, in production workloads using resourcesPreset
is discouraged as it may not fully adapt to your specific needs. Find more information on container resource management in the official Kubernetes documentation.
It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.
Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.
This chart can be integrated with Prometheus by setting metrics.enabled
to true. This will expose the etcd native Prometheus port in the container and service (if metrics.useSeparateEndpoint=true
). It will all have the necessary annotations to be automatically scraped by Prometheus.
Prometheus requirements
It is necessary to have a working installation of Prometheus or Prometheus Operator for the integration to work. Install the Bitnami Prometheus helm chart or the Bitnami Kube Prometheus helm chart to easily have a working Prometheus in your cluster.
Integration with Prometheus Operator
The chart can deploy PodMonitor
objects for integration with Prometheus Operator installations. To do so, set the value *.metrics.podMonitor.enabled=true
. Ensure that the Prometheus Operator CustomResourceDefinitions
are installed in the cluster or it will fail with the following error:
no matches for kind "PodMonitor" in version "monitoring.coreos.com/v1"
Install the Bitnami Kube Prometheus helm chart for having the necessary CRDs and the Prometheus Operator.
Bitnami charts configure credentials at first boot. Any further change in the secrets or credentials require manual intervention. Follow these instructions:
kubectl create secret generic SECRET_NAME --from-literal=etcd-root-password=PASSWORD --dry-run -o yaml | kubectl apply -f -
The Bitnami etcd chart can be used to bootstrap an etcd cluster, easy to scale and with available features to implement disaster recovery. It uses static discovery configured via environment variables to bootstrap the etcd cluster. Based on the number of initial replicas, and using the A records added to the DNS configuration by the headless service, the chart can calculate every advertised peer URL.
The chart makes use of some extra elements offered by Kubernetes to ensure the bootstrapping is successful:
Learn more about etcd discovery, Pod Management Policies and recording "not ready" pods.
Here is an example of the environment configuration bootstrapping an etcd cluster with 3 replicas:
Member | Variable | Value |
---|---|---|
0 | ETCD_NAME | etcd-0 |
0 | ETCD_INITIAL_ADVERTISE_PEER_URLS | http://etcd-0.etcd-headless.default.svc.cluster.local:2380 |
--------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
1 | ETCD_NAME | etcd-1 |
1 | ETCD_INITIAL_ADVERTISE_PEER_URLS | http://etcd-1.etcd-headless.default.svc.cluster.local:2380 |
--------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
2 | ETCD_NAME | etcd-2 |
2 | ETCD_INITIAL_ADVERTISE_PEER_URLS | http://etcd-2.etcd-headless.default.svc.cluster.local:2380 |
--------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
* | ETCD_INITIAL_CLUSTER_STATE | new |
* | ETCD_INITIAL_CLUSTER_TOKEN | etcd-cluster-k8s |
* | ETCD_INITIAL_CLUSTER | etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380 |
The probes (readiness & liveness) are delayed 60 seconds by default, to give the etcd replicas time to start and find each other. After that period, the etcdctl endpoint health command is used to periodically perform health checks on every replica.
Scalability
The Bitnami etcd chart uses etcd reconfiguration operations to add/remove members of the cluster during scaling.
When scaling down, a "pre-stop" lifecycle hook is used to ensure that the etcdctl member remove
command is executed. The hook stores the output of this command in the persistent volume attached to the etcd pod. This hook is also executed when the pod is manually removed using the kubectl delete pod
command or rescheduled by Kubernetes for any reason. This implies that the cluster can be scaled up/down without human intervention.
Here is an example to explain how this works:
If, for whatever reason, the "pre-stop" hook fails at removing the member, the initialization logic is able to detect that something went wrong by checking the etcdctl member remove
command output that was stored in the persistent volume. It then uses the etcdctl member update
command to add back the member. In this case, the cluster isn't automatically scaled down/up while the pod is recovered. Therefore, when other members attempt to connect to the pod, it may cause warnings or errors like the one below:
E | rafthttp: failed to dial XXXXXXXX on stream Message (peer XXXXXXXX failed to find local node YYYYYYYYY)
I | rafthttp: peer XXXXXXXX became inactive (message send to peer failed)
W | rafthttp: health check for peer XXXXXXXX could not connect: dial tcp A.B.C.D:2380: i/o timeout
Learn more about etcd runtime configuration and how to safely drain a Kubernetes node.
Cluster updates
When updating the etcd StatefulSet (such as when upgrading the chart version via the helm upgrade command), every pod must be replaced following the StatefulSet update strategy.
The chart uses a "RollingUpdate" strategy by default and with default Kubernetes values. In other words, it updates each Pod, one at a time, in the same order as Pod termination (from the largest ordinal to the smallest). It will wait until an updated Pod is "Running" and "Ready" prior to updating its predecessor.
Learn more about StatefulSet update strategies.
Disaster recovery
If, for whatever reason, (N-1)/2 members of the cluster fail and the "pre-stop" hooks also fail at removing them from the cluster, the cluster disastrously fails, irrevocably losing quorum. Once quorum is lost, the cluster cannot reach consensus and therefore cannot continue accepting updates. Under this circumstance, the only possible solution is usually to restore the cluster from a snapshot.
IMPORTANT: All members should restore using the same snapshot.
The Bitnami etcd chart solves this problem by optionally offering a Kubernetes cron job that periodically snapshots the keyspace and stores it in a RWX volume. In case the cluster disastrously fails, the pods will automatically try to restore it using the last avalable snapshot.
Learn how to enable this disaster recovery feature.
The chart also sets by default a "soft" Pod AntiAffinity to reduce the risk of the cluster failing disastrously.
Learn more about etcd recovery, Kubernetes cron jobs and pod affinity and anti-affinity
The etcd chart can be configured with Role-based access control and TLS encryption to improve its security.
Configure RBAC
In order to enable Role-Based Access Control for etcd, set the following parameters:
auth.rbac.create=true
auth.rbac.rootPassword=ETCD_ROOT_PASSWORD
These parameters create a root
user with an associate root
role with access to everything. The remaining users will use the guest
role and won't have permissions to do anything.
Configure TLS for server-to-server communications
In order to enable secure transport between peer nodes deploy the helm chart with these options:
auth.peer.secureTransport=true
auth.peer.useAutoTLS=true
Configure certificates for client communication
In order to enable secure transport between client and server, create a secret containing the certificate and key files and the CA used to sign the client certificates. In this case, create the secret and then deploy the chart with these options:
auth.client.secureTransport=true
auth.client.enableAuthentication=true
auth.client.existingSecret=etcd-client-certs
Learn more about the etcd security model and how to generate self-signed certificates for etcd.
The Bitnami etcd Helm chart supports automatic disaster recovery by periodically snapshotting the keyspace. If the cluster permanently loses more than (N-1)/2 members, it tries to recover the cluster from a previous snapshot.
Enable this feature with the following parameters:
persistence.enabled=true
disasterRecovery.enabled=true
disasterRecovery.pvc.size=2Gi
disasterRecovery.pvc.storageClassName=nfs
If the startFromSnapshot.*
parameters are used at the same time as the disasterRecovery.*
parameters, the PVC provided via the startFromSnapshot.existingClaim
parameter will be used to store the periodical snapshots.
NOTE: The disaster recovery feature requires volumes with ReadWriteMany access mode.
Two different approaches are available to back up and restore this Helm Chart:
Method 1: Backup and restore data using etcd's built-in tools
This method involves the following steps:
NOTE: Under this approach, it is important to create the new deployment on the destination cluster using the same credentials as the original deployment on the source cluster.
Method 2: Back up and restore persistent data volumes
This method involves copying the persistent data volumes for the etcd nodes and reusing them in a new deployment with Velero, an open source Kubernetes backup/restore tool. This method is only suitable when:
This method involves the following steps:
The metrics exposed by etcd can be exposed to be scraped by Prometheus. Metrics can be scraped from within the cluster using any of the following approaches:
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics/cluster"
prometheus.io/port: "9000"
If metrics are to be scraped from outside the cluster, the Kubernetes API proxy can be utilized to access the endpoint.
In order to use custom configuration parameters, two options are available:
extraEnvVars
property. Alternatively, you can use a ConfigMap or a Secret with the environment variables using the extraEnvVarsCM
or the extraEnvVarsSecret
properties.extraEnvVars:
- name: ETCD_AUTO_COMPACTION_RETENTION
value: "0"
- name: ETCD_HEARTBEAT_INTERVAL
value: "150"
etcd.conf.yml
: The etcd chart allows mounting a custom etcd.conf.yml
file as ConfigMap. In order to so, you can use the configuration
property. Alternatively, you can use an existing ConfigMap using the existingConfigmap
parameter.Since etcd keeps an exact history of its keyspace, this history should be periodically compacted to avoid performance degradation and eventual storage space exhaustion. Compacting the keyspace history drops all information about keys superseded prior to a given keyspace revision. The space used by these keys then becomes available for additional writes to the keyspace.
autoCompactionMode
, by default periodic. Valid values: "periodic", "revision".
autoCompactionRetention
for mvcc key value store in hour, by default 0, means disabled.You can enable auto compaction by using following parameters:
autoCompactionMode=periodic
autoCompactionRetention=10m
If you have a need for additional containers to run within the same pod as the etcd app (e.g. an additional metrics or logging exporter), you can do so via the sidecars
config parameter. Simply define your container according to the Kubernetes container spec.
sidecars:
- name: your-image-name
image: your-image
imagePullPolicy: Always
ports:
- name: portname
containerPort: 1234
Similarly, you can add extra init containers using the initContainers
parameter.
initContainers:
- name: your-image-name
image: your-image
imagePullPolicy: Always
ports:
- name: portname
containerPort: 1234
There are cases where you may want to deploy extra objects, such a ConfigMap containing your app's configuration or some extra deployment with a micro service used by your app. For covering this case, the chart allows adding the full specification of other objects using the extraDeploy
parameter.
This chart allows you to set your custom affinity using the affinity
parameter. Find more information about Pod's affinity in the kubernetes documentation.
As an alternative, you can use of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common chart. To do so, set the podAffinityPreset
, podAntiAffinityPreset
, or nodeAffinityPreset
parameters.
The Bitnami etcd image stores the etcd data at the /bitnami/etcd
path of the container. Persistent Volume Claims are used to keep the data across statefulsets.
The chart mounts a Persistent Volume volume at this location. The volume is created using dynamic volume provisioning by default. An existing PersistentVolumeClaim can also be defined for this purpose.
If you encounter errors when working with persistent volumes, refer to our troubleshooting guide for persistent volumes.
| Name | Description | Value | | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Note: the README for this chart is longer than the DockerHub length limit of 25000, so it has been trimmed. The full README can be found at https://github.com/bitnami/charts/blob/main/bitnami/etcd/README.md