Bitnami package for Fluentd

Fluentd collects events from various data sources and writes them to files, RDBMS, NoSQL, IaaS, SaaS, Hadoop and so on.

Overview of Fluentd⁠

Trademarks: This software listing is packaged by Bitnami. The respective trademarks mentioned in the offering are owned by the respective companies, and use of them does not imply any affiliation or endorsement.

TL;DR

helm install my-release oci://registry-1.docker.io/bitnamicharts/fluentd

Looking to use Fluentd in production? Try VMware Tanzu Application Catalog⁠, the commercial edition of the Bitnami catalog.

Introduction

This chart bootstraps a Fluentd⁠ deployment on a Kubernetes⁠ cluster using the Helm⁠ package manager.

Bitnami charts can be used with Kubeapps⁠ for deployment and management of Helm Charts in clusters.

Prerequisites

• Kubernetes 1.23+
• Helm 3.8.0+
• PV provisioner support in the underlying infrastructure

Installing the Chart

To install the chart with the release name my-release:

helm install my-release oci://REGISTRY_NAME/REPOSITORY_NAME/fluentd

These commands deploy Fluentd on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Configuration and installation details

Resource requests and limits

Bitnami charts allow setting resource requests and limits for all containers inside the chart deployment. These are inside the resources value (check parameter table). Setting requests is essential for production workloads and these should be adapted to your specific use case.

To make this process easier, the chart contains the resourcesPreset values, which automatically sets the resources section according to different presets. Check these presets in the bitnami/common chart⁠. However, in production workloads using resourcesPreset is discouraged as it may not fully adapt to your specific needs. Find more information on container resource management in the official Kubernetes documentation⁠.

Rolling VS Immutable tags⁠

It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.

Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.

Forwarding the logs to another service

By default, the aggregators in this chart will send the processed logs to the standard output. However, a common practice is to send them to another service, like Elasticsearch, instead. This can be achieved with this Helm Chart by mounting your own configuration files. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-output
data:
  fluentd.conf: |
    # Prometheus Exporter Plugin
    # input plugin that exports metrics
    <source>
      @type prometheus
      port 24231
    </source>

    # input plugin that collects metrics from MonitorAgent
    <source>
      @type prometheus_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

    # input plugin that collects metrics for output plugin
    <source>
      @type prometheus_output_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

    # Ignore fluentd own events
    <match fluent.**>
      @type null
    </match>

    # TCP input to receive logs from the forwarders
    <source>
      @type forward
      bind 0.0.0.0
      port 24224
    </source>

    # HTTP input for the liveness and readiness probes
    <source>
      @type http
      bind 0.0.0.0
      port 9880
    </source>

    # Throw the healthcheck to the standard output instead of forwarding it
    <match fluentd.healthcheck>
      @type stdout
    </match>

    # Send the logs to the standard output
    <match **>
      @type elasticsearch
      include_tag_key true
      host "#{ENV['ELASTICSEARCH_HOST']}"
      port "#{ENV['ELASTICSEARCH_PORT']}"
      logstash_format true

      <buffer>
        @type file
        path /opt/bitnami/fluentd/logs/buffers/logs.buffer
        flush_thread_count 2
        flush_interval 5s
      </buffer>
    </match>

As an example, using the above configmap, you should specify the required parameters when upgrading or installing the chart:

aggregator.configMap=elasticsearch-output
aggregator.extraEnvVars[0].name=ELASTICSEARCH_HOST
aggregator.extraEnvVars[0].value=your-ip-here
aggregator.extraEnvVars[1].name=ELASTICSEARCH_PORT
aggregator.extraEnvVars[1].value=your-port-here

Prometheus metrics

This chart can be integrated with Prometheus by setting metrics.enabled to true. This will expose Fluentd native Prometheus endpoint in a metrics service that can be configured under the metrics.service section. It will have the necessary annotations to be automatically scraped by Prometheus.

Prometheus requirements

It is necessary to have a working installation of Prometheus or Prometheus Operator for the integration to work. Install the Bitnami Prometheus helm chart⁠ or the Bitnami Kube Prometheus helm chart⁠ to easily have a working Prometheus in your cluster.

Integration with Prometheus Operator

The chart can deploy ServiceMonitor objects for integration with Prometheus Operator installations. To do so, set the value metrics.serviceMonitor.enabled=true. Ensure that the Prometheus Operator CustomResourceDefinitions are installed in the cluster or it will fail with the following error:

no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

Install the Bitnami Kube Prometheus helm chart⁠ for having the necessary CRDs and the Prometheus Operator.

Backup and restore

To back up and restore Helm chart deployments on Kubernetes, you need to back up the persistent volumes from the source deployment and attach them to a new deployment using Velero⁠, a Kubernetes backup/restore tool. Find the instructions for using Velero in this guide⁠.

Using custom init scripts

For advanced operations, the Bitnami Fluentd charts allows using custom init scripts that will be mounted inside /docker-entrypoint.init-db. You can include the file directly in your values.yaml, depending on where you are going to initialize your scripts with aggregator.initScripts (or forwarder.initScripts), or use a ConfigMap or a Secret (in case of sensitive data) for mounting these extra scripts. In this case you use the aggregator.initScriptsCM and aggregator.initScriptsSecret values (the same for forwarder).

initScriptsCM=special-scripts
initScriptsSecret=special-scripts-sensitive

Forwarder Security Context & Policy

By default, the forwarderDaemonSet from this chart runs as the root user, within the root group, assigning root file system permissions. This is different to the default behaviour of most Bitnami Helm charts where we prefer to work with non-root containers⁠.

The default behaviour is to run as root because:

• the forwarder needs to mount hostPath volumes from the underlying node to read Docker container (& potentially other) logs
• in many Kubernetes node distributions, these log files are not readable by anyone other than root
• fsGroup doesn't work with hostPath volumes to allow the process to run non-root with alternate file system permissions

Since we would like the chart to work out-of-the-box for as many users as possible, the forwarder thus runs as root by default. You can read more about the motivation for this at #1905⁠ and #2323⁠, however you should be aware of this, and the risks of running root containers in general.

If you enable the forwarder's bundled PodSecurityPolicy⁠ with forwarder.rbac.pspEnabled=true it will allow the pod to run as root by default, while ensuring as many other privileges as possible are dropped.

Running as non-root

You can run as the fluentd user/group (non-root) with the below overrides if:

• you have control of the hostPath filesystem permissions on your nodes sufficient to allow the fluentd user to read from them
• don't need to write to the hostPaths

Note that if you have enabled the bundled PodSecurityPolicy⁠, it will adapt to the Chart values overrides.

forwarder:
  daemonUser: fluentd
  daemonGroup: fluentd

  securityContext:
    runAsUser: 1001
    runAsGroup: 1001
    fsGroup: 1001

Pod Security Policy & Custom hostPaths

Mounting additional hostPaths is sometimes required to deal with /var/lib being symlinked on some Kubernetes environments. If you need to do so, the bundled PodSecurityPolicy⁠ will likely not meet your needs, as it whitelists only the standard hostPaths.

Setting Pod's affinity

This chart allows you to set your custom affinity using the XXX.affinity parameter(s). Find more information about Pod's affinity in the kubernetes documentation⁠.

As an alternative, you can use of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common⁠ chart. To do so, set the XXX.podAffinityPreset, XXX.podAntiAffinityPreset, or XXX.nodeAffinityPreset parameters.

Securing traffic using TLS

TLS for the Fluentd can be enabled by setting tls.enabled=true. The chart allows two configuration options:

• Provide your own secrets for the forwarder and aggregator using the tls.aggregator.existingSecret and tls.forwarder.existingSecret values.
• Have the chart auto-generate the certificates using tls.autoGenerated=true.

Parameters

Global parameters

Common parameters

Fluentd parameters

Note: the README for this chart is longer than the DockerHub length limit of 25000, so it has been trimmed. The full README can be found at https://github.com/bitnami/charts/blob/main/bitnami/fluentd/README.md⁠