atlassianlabs/fluentd repository overview

⁠Overview

fluentd docker image with ruby 2.2.x and support for both elasticsearch and kinesis

⁠Configuration

Create a fluentd.conf file following fluentd's configuration file format. Add any input that you wish to and add the kinesis output (see example file in this repository).

⁠Installed plugins

The following plugins are installed in the docker image:

1. fluent-plugin-ec2-metadata
2. fluent-plugin-hostname
3. fluent-plugin-retag
4. fluent-plugin-kinesis
5. fluent-plugin-kinesis-aggregation
6. fluent-plugin-elasticsearch
7. fluent-plugin-record-modifier
8. fluent-plugin-multi-format-parser

Here is an example configuration to use them. Note that this uses the old 'match/retag' approach; ideally you should use filter plugins like record-modifier (see fluentd.conf), but unfortunately there is no filter equivalent for ec2metadata yet.

<match syslog>
  type hostname
  key_name ec2_hostname
  add_prefix hostname
</match>
<match hostname.syslog>
  type ec2_metadata
  output_tag ec2.${tag}
  <record>
    ec2_instance_id   ${instance_id}
    ec2_instance_type ${instance_type}
    ec2_az            ${availability_zone}
    service_id        my-service-uuid
    env               my-environment
  </record>
</match>
<match ec2.hostname.**>
  type retag
  remove_prefix ec2.hostname
</match>

⁠Run

Mount the volumes where your logs are if needed and the path to the fluentd configuration file. Pass aws credentials via environment variables:

docker run --ulimit nofile=65536:65536 -p 24224 -p 24220 -p 24230 \
        -v /var/log:/fluentd/log -v `pwd`:/etc/fluent \
        atlassianlabs/fluentd:0.5.1