Public | Automated Build

Last pushed: a month ago
Short Description
Short description is empty for this repo.
Full Description

Docker-Fluentd: the Container to Log Other Containers' Logs

What

By running this container with the following command, one can aggregate the logs of Docker containers running on the same host:

docker run -d -v /var/lib/docker/containers:/var/lib/docker/containers kiyoto/docker-fluentd

By default, the container logs are stored in /var/log/docker/yyyyMMdd.log inside this logging container. The data is buffered, so you may also see buffer files like /var/log/docker/20141114.b507c71e6fe540eab.log where "b507c71e6fe540eab" is a hash identifier. You can mount that container volume back to host. Also, by modifying fluent.conf and rebuilding the Docker image, you can stream up your logs to Elasticsearch, Amazon S3, MongoDB, Treasure Data, etc.

The output log looks exactly like Docker's JSON formatted logs, except each line also has the "container_id" field:

{"log":"Fri Nov 14 01:03:19 UTC 2014\r\n","stream":"stdout","container_id":"db18480728ed247a64bf6df49684cb246a38bbe11f14276d4c2bb84f56255ff4"}

How

docker-fluentd uses Fluentd inside to tail log files that are mounted on /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log. It uses the tail input plugin to tail JSON-formatted log files that each Docker container emits.

Then, Fluentd adds the "container_id" field using the record reformer before writing out to local files using the file output plugin.

Fluentd has a lot of plugins and can probably support your data output destination. For example, if you want to output your logs to Elasticsearch instead of files, then, add the following lines to Dockerfile

RUN ["apt-get", "update"]
RUN ["apt-get", "install", "--yes", "make", "libcurl4-gnutls-dev"]
RUN ["/usr/local/bin/gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri"]

right before "ENTRYPOINT". This installs the output plugin for Elasticsearch. Then, update fluent.conf as follows.

<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S 
  tag docker.*
  format json
</source>

<match docker.var.lib.docker.containers.*.*.log>
  type record_reformer
  container_id ${tag_parts[5]}
  tag docker.all
</match>

<match docker.all>
  type elasticsearch
  log_level info
  host YOUR_ES_HOST
  port YOUR_ES_PORT
  include_tag_key true 
  logstash_format true
  flush_intercal 5s
</match>

What's Next?

Docker Pull Command
Owner
kiyoto
Source Repository

Comments (2)
nghiaht
2 years ago

The fluentd logging driver is more simple when you run Docker directly by docker run commands. If you do, it require an existing fluentd host or docker run would fail.
This container separate that thing, you can leave your docker containers run on their way, and logging may happen later with this container.

telamon
2 years ago

What's the downside to use this container in comparison to the fluentd logging driver ?