Sign inSign up

picoded/docker-filebeat

By picoded

Updated about 8 years ago

Pushing of normalised docker logs, to elasticsearch

Image
0

10K+

picoded/docker-filebeat repository overview

picoded/docker-filebeat

https://hub.docker.com/r/picoded/docker-filebeat/ https://github.com/picoded/dockerfiles/tree/master/logging/docker-filebeat/

Summary

Pushing of normalised docker logs, to elasticsearch

This was designed to work with logrotate for long term usage.

https://hub.docker.com/r/picoded/docker-logrotate/
https://github.com/picoded/dockerfiles/tree/master/logging/docker-logrotate/

You will need the following volume mounted, with read/write access.
/var/lib/docker/containers/:/var/lib/docker/containers/:rw
/var/lib/filebeat/:/var/lib/filebeat/:rw

And of course elasticsearch configured.

Note this is NOT designed for 0 second realtime log pushes. (Beside you cant search so in Elasticsearch).

Configuration

# Log path to read from docker containers
ENV LOG_PATH     "/var/lib/docker/containers/*/*-json.log"

# Regex rules to filter out lines
# This is useful for extremly noisy containers,
# With little logging value : Such as rancher ipsec-router
#
# These filters were however carefully chosen, to avoid accidentally blocking off
# logs that could have been wanted elsewhere. In most cases that is.
# Modify these filters as you require.
ENV LOG_EXCLUDE_LINE ".*(([0-9]{2}\[(KNL|IKE|MGR|NET)\])|(POST /_bulk HTTP/1.1\" 200 [0-9]* \"-\" \"Go-http-client/1.1)|(level\=debug.*io\.rancher)).*"

# A single log line maximum bytes
# 900,000 bytes was selected, so that 10 log lines
# would be under 10MB with some format overhead.
ENV LOG_LINE_MAX 900000

# Elasticsearch host to push into
ENV ES_HOST    elasticsearch

# Elasticsearch port to push into
ENV ES_PORT    9200

# Elasticsearch index to use 
ENV ES_INDEX  "docker-filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

# Compression level of log submissions 0 - 9
# NOTE: This is not supported in AWS Elasticsearch
ENV ES_COMPRESSION 0

# Bulk batch size. This is used in conjuncture with LOG_LINE_MAX
# To ensure that the total BULK request size does not exceed 10MB
ENV ES_BATCHSIZE 10

Issue filling

Any problems / comments / kudos should be filed at github =) https://github.com/picoded/dockerfiles

Tag summary

Content type

Image

Digest

Size

97.1 MB

Last updated

about 8 years ago

docker pull picoded/docker-filebeat