Public | Automated Build

Last pushed: 2 days ago
Short Description
Fluentd Docker image by fluent project: http://www.fluentd.org/
Full Description

Fluentd Docker Image





Supported tags and respective Dockerfile links

Using Kubernetes?

Check fluentd-kubernetes-daemonset images.

What is Fluentd?

Fluentd is an open source data collector, which lets you unify the data
collection and consumption for a better use and understanding of data.

www.fluentd.org

How to use this image

To create endpoint that collectc logs on your host just run:

docker run -d -p 24224:24224 -p 24224:24224/udp -v /data:/fluentd/log fluent/fluentd

Default configurations are to:

  • listen port 24224 for Fluentd forward protocol
  • store logs with tag docker.** into /fluentd/log/docker.*.log
    (and symlink docker.log)
  • store all other logs into /fluentd/log/data.*.log (and symlink data.log)

Environment Variables

Environment variable below are configurable to control how to execute fluentd process:

FLUENTD_CONF

This variable allows you to specify configuration file name that will be used
in -c Fluentd command line option.

If you want to use your own configuration file (without any optional plugins),
you can do it with this environment variable and Docker volumes (-v option
of docker run).

  1. Write configuration file with filename yours.conf.
  2. Execute docker run with -v /path/to/dir:/fluentd/etc
    to share /path/to/dir/yours.conf in container,
    and -e FLUENTD_CONF=yours.conf to read it.

FLUENTD_OPT

Use this variable to specify other Fluentd command line options,
like -v or -q.

FLUENT_UID

Use this variable to specify user id of fluent user.

Image versions

This image is based on the popular Alpine Linux project, available in
the alpine official image.
Alpine Linux is much smaller than most distribution base images (~5MB), and
thus leads to much slimmer images in general.

Since v0.12.26, tags are separated into vX.Y.Z and vX.Y.Z-onbuild.

stable, latest

Latest version of stable Fluentd branch (currently v0.12).

edge

Latest version of edge Fluentd branch (currently v0.14).

vX.Y

Latest version of vX.Y Fluentd branch.

vX.Y.Z

Concrete vX.Y.Z version of Fluentd.

onbuild, xxx-onbuild

This image makes building derivative images easier.
See "How to build your own image" section for
more details.

debian

The image based on Debian Linux image.
You may use this image when you require plugins which cannot be installed
on Alpine (like fluent-plugin-systemd).

How to build your own image

You can build a customized image based on Fluentd's onbuild image.
Customized image can include plugins and fluent.conf file.

1. Create a working directory

We will use this directory to build a Docker image.
Type following commands on a terminal to prepare a minimal project first:

# Create project directory.
mkdir custom-fluentd
cd custom-fluentd

# Download default fluent.conf. This file will be copied to the new image.
# VERSION is v0.12 or v0.14 like fluentd version and OS is alpine or debian.
# Full example is https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/v0.12/debian-onbuild/fluent.conf
curl https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/VERSION/OS-onbuild/fluent.conf > fluent.conf

# Create plugins directory. plugins scripts put here will be copied to the new image.
mkdir plugins

# Download sample Dockerfile. If you use v0.14.15/v0.12.34 or earlier image, use Dockerfile.sample.old
curl https://raw.githubusercontent.com/fluent/fluentd-docker-image/master/Dockerfile.sample > Dockerfile

2. Customize fluent.conf

Documentation of fluent.conf is available at docs.fluentd.org.

3. Customize Dockerfile to install plugins (optional)

You can install Fluentd plugins using Dockerfile.
Sample Dockerfile installs fluent-plugin-elasticsearch.
To add plugins, edit Dockerfile as following:

Alpine version
  • Latest and v0.14.16/v0.12.35 or later
# or v0.14-onbuild
FROM fluent/fluentd:v0.12-onbuild

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish

RUN apk add --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo gem install \
        fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /var/cache/apk/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem
  • v0.14.15/v0.12.34 or earlier

Need USER line.

# or v0.14-onbuild
FROM fluent/fluentd:v0.12-onbuild

USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish

RUN apk add --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && sudo -u fluent gem install \
        fluent-plugin-elasticsearch \
        fluent-plugin-record-reformer \
 && sudo -u fluent gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /var/cache/apk/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem

USER fluent
Debian version
  • Latest and v0.14.16/v0.12.35 or later
# or v0.14-debian-onbuild
FROM fluent/fluentd:v0.12-debian-onbuild

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install \
        fluent-plugin-elasticsearch \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem
  • v0.14.15/v0.12.34 or earlier

Need USER line.

# or v0.14-debian-onbuild
FROM fluent/fluentd:v0.12-debian-onbuild

USER root

# below RUN includes plugin as examples elasticsearch is not required
# you may customize including plugins as you wish

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo -u fluent gem install \
        fluent-plugin-elasticsearch \
 && sudo -u fluent gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem

USER fluent
Note

These example run apk add/apt-get install to be able to install
Fluentd plugins which require native extensions (they are removed immediately
after plugin installation).
If you're sure that plugins don't include native extensions, you can omit it
to make image build faster.

4. Build image

Use docker build command to build the image.
This example names the image as custom-fluentd:latest:

docker build -t custom-fluentd:latest ./

5. Test it

Once the image is built, it's ready to run.
Following commands run Fluentd sharing ./log directory with the host machine:

mkdir -p log
docker run -it --rm --name custom-docker-fluent-logger -v $(pwd)/log:/fluentd/log custom-fluentd:latest

Open another terminal and type following command to inspect IP address.
Fluentd is running on this IP address:

docker inspect -f '{{.NetworkSettings.IPAddress}}' custom-docker-fluent-logger

Let's try to use another docker container to send its logs to Fluentd.

docker run --log-driver=fluentd --log-opt tag="docker.{{.ID}}" --log-opt fluentd-address=FLUENTD.ADD.RE.SS:24224 python:alpine echo Hello
# and force flush buffered logs
docker kill -s USR1 custom-docker-fluent-logger

(replace FLUENTD.ADD.RE.SS with actual IP address you inspected at
the previous step)

You will see some logs sent to Fluentd.

References

Docker Logging | fluentd.org

Fluentd logging driver - Docker Docs

Issues

We can't notice comments in the DockerHub so don't use them for reporting issue
or asking question.

If you have any problems with or questions about this image, please contact us
through a GitHub issue.

Docker Pull Command
Owner
fluent
Source Repository

Comments (10)
pedroarthur
2 months ago

Thanks, folks!

In the curl commands above, you can use -o filename (ie, -o Dockerfile) intead of the output redirection operator.

rajkirangade
a year ago

@activatedgeek
The docker logscommand is available only for the json-file and journald logging drivers.

repeatedly
a year ago

Sorry we can't notice the comment in the DockerHub so don't use this comment for reporting issue or asking question.
Use github repository or fluentd mailing list instead: https://github.com/fluent/fluentd-docker-image / https://groups.google.com/forum/#!forum/fluentd

apollov
a year ago

This start script helped me to finally make plugin work:

#!/bin/sh

export PATH=/home/fluent/.gem/ruby/2.3.0/bin:$PATH
export GEM_PATH=/home/fluent/.gem/ruby/2.3.0:$GEM_$PATH
fluent-gem install fluent-plugin-elasticsearch
exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT
activatedgeek
a year ago

Doesn't the logging driver work with docker-compose?

I get the following error: "logs" command is supported only for "json-file" and "journald" logging drivers (got: fluentd)

jesaja
a year ago

Shouldn't this container print log messages to stdout due to the stdout_filter? For me the log messages end up in the files in /fluentd/log/. But except the output of fluentd itself nothing is written to stdout in the container.
I'm not familiar with fluentd, maybe I understand something wrong?

repeatedly
2 years ago

Thanks for the comment. I just added tags. I thought DockerHub builds the image for each github tag automatically...

sstiehl
2 years ago

+1 for tagging

robertfirek
2 years ago

I suggest to add version-specific tags. Dockerfile on this page is out of data and it doesn't show the latest version of fluentd installed in the image.

chiefy
2 years ago

Can you tag specific versions (not just 'latest')? Thanks!