Bitnami package for Logstash

Logstash is an open source data processing engine. It ingests data from multiple sources, processes it, and sends the output to final destination in real-time. It is a core component of the ELK stack.

Overview of Logstash⁠

Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.

TL;DR

helm install my-release oci://registry-1.docker.io/bitnamicharts/logstash

Looking to use Logstash in production? Try VMware Tanzu Application Catalog⁠, the commercial edition of the Bitnami catalog.

Introduction

This chart bootstraps a logstash⁠ deployment on a Kubernetes⁠ cluster using the Helm⁠ package manager.

Bitnami charts can be used with Kubeapps⁠ for deployment and management of Helm Charts in clusters.

Prerequisites

• Kubernetes 1.23+
• Helm 3.8.0+

Installing the Chart

To install the chart with the release name my-release:

helm install my-release oci://REGISTRY_NAME/REPOSITORY_NAME/logstash

These commands deploy logstash on the Kubernetes cluster in the default configuration. The configuration section lists the parameters that can be configured during installation.

Configuration and installation details

Resource requests and limits

Bitnami charts allow setting resource requests and limits for all containers inside the chart deployment. These are inside the resources value (check parameter table). Setting requests is essential for production workloads and these should be adapted to your specific use case.

To make this process easier, the chart contains the resourcesPreset values, which automatically sets the resources section according to different presets. Check these presets in the bitnami/common chart⁠. However, in production workloads using resourcesPreset is discouraged as it may not fully adapt to your specific needs. Find more information on container resource management in the official Kubernetes documentation⁠.

Rolling vs Immutable tags⁠

It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.

Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.

Expose the Logstash service

The service(s) created by the deployment can be exposed within or outside the cluster using any of the following approaches:

• Ingress: This requires an Ingress controller to be installed in the Kubernetes cluster. Set ingress.enabled=true to expose the corresponding service(s) through Ingress.
• ClusterIP: This exposes the service(s) on a cluster-internal IP address. This approach makes the corresponding service(s) reachable only from within the cluster. Set service.type=ClusterIP to choose this approach.
• NodePort: This exposes the service() on each node's IP address at a static port (the NodePort). This approach makes the corresponding service(s) reachable from outside the cluster by requesting the static port using the node's IP address, such as NODE-IP:NODE-PORT. Set service.type=NodePort to choose this approach.
• LoadBalancer: This exposes the service(s) externally using a cloud provider's load balancer. Set service.type=LoadBalancer to choose this approach.

Use custom configuration

By default, this Helm chart provides a basic configuration for Logstash: listening to HTTP requests on port 8080 and writing them to the standard output.

This Logstash configuration can be adjusted using the input, filter, and output parameters, which allow specification of the input, filter and output plugins configuration respectively. In addition to these options, the chart also supports reading configuration from an external ConfigMap via the existingConfiguration parameter. Note that this will override the parameters discussed previously.

Create and use multiple pipelines

The chart supports the use of multiple pipelines⁠ by setting the enableMultiplePipelines parameter to true.

To do this, place the pipelines.yml file in the files/conf directory, together with the rest of the desired configuration files. If the enableMultiplePipelines parameter is set to true but the pipelines.yml file does not exist in the mounted volume, a dummy file is created using the default configuration (a single pipeline).

The chart also supports setting an external ConfigMap with all the configuration filesvia the existingConfiguration parameter.

Here is an example of creating multiple pipelines using a ConfigMap:

• Create a ConfigMap with the configuration files:

  $ cat bye.conf
  input {
    file {
      path => "/tmp/bye"
    }
  }
  output {
    stdout { }
  }
  
  $ cat hello.conf
  input {
    file {
      path => "/tmp/hello"
    }
  }
  output {
    stdout { }
  }
  
  $ cat pipelines.yml
  - pipeline.id: hello
    path.config: "/opt/bitnami/logstash/config/hello.conf"
  - pipeline.id: bye
    path.config: "/opt/bitnami/logstash/config/bye.conf"
  
  $ kubectl create cm multipleconfig --from-file=pipelines.yml --from-file=hello.conf --from-file=bye.conf
  

  Copy

• Deploy the Helm chart with the enableMultiplePipelines parameter:

  helm install logstash . --set enableMultiplePipelines=true --set existingConfiguration=multipleconfig
  

  Copy

• Create dummy events in the tracked files and check the result in the Logstash output:

  kubectl exec -ti logstash-0 -- bash -c 'echo hi >> /tmp/hello'
  kubectl exec -ti logstash-0 -- bash -c 'echo bye >> /tmp/bye'
  

  Copy

Add extra environment variables

To add extra environment variables, use the extraEnvVars property.

extraEnvVars:
  - name: ELASTICSEARCH_HOST
    value: "x.y.z"

To add extra environment variables from an external ConfigMap or secret, use the extraEnvVarsCM and extraEnvVarsSecret properties. Note that the secret and ConfigMap should be already available in the namespace.

extraEnvVarsSecret: logstash-secrets
extraEnvVarsCM: logstash-configmap

Set Pod affinity

This chart allows you to set custom Pod affinity using the affinity parameter. Find more information about Pod affinity in the kubernetes documentation⁠.

As an alternative, use one of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common⁠ chart. To do so, set the podAffinityPreset, podAntiAffinityPreset, or nodeAffinityPreset parameters.

Backup and restore

To back up and restore Helm chart deployments on Kubernetes, you need to back up the persistent volumes from the source deployment and attach them to a new deployment using Velero⁠, a Kubernetes backup/restore tool. Find the instructions for using Velero in this guide⁠.

Persistence

The Bitnami Logstash⁠ image stores the Logstash data at the /bitnami/logstash/data path of the container.

Persistent Volume Claims (PVCs) are used to keep the data across deployments. This is known to work in GCE, AWS, and minikube.

See the Parameters section to configure the PVC or to disable persistence.

Parameters

Global parameters

Common parameters

Logstash parameters

Note: the README for this chart is longer than the DockerHub length limit of 25000, so it has been trimmed. The full README can be found at https://github.com/bitnami/charts/blob/main/bitnami/logstash/README.md⁠