Docker Logstash Integration
- any Docker log files
- whatever log files you configure logstash-forwarder to ship within a container (just put a config at
/etc/logstash-forwarder.conf, only the
filessection gets evaluated while
networksection is globally configured).
I wasn't too happy with existing possibilities and while I know that the Docker team is working on a solution, this scratches my itch right now.
Also I didn't see an obvious way to extend docker-gen to handle generic in container templates.
Besides that, how much reason do you need to play with Go & Docker? ;-)
How it works:
docker-logstash-forwarder listens to Docker events and continually restarts a logstash-forwarder instance, after refreshing its configuration, every
laziness seconds after a new event was received (to avoid unnecessary restarts - configurable via
-laziness flag - defaults to 5 seconds).
logstash-forwarder will ignore containers that match a given regex passed in via the command line with -ignore-regexp-from or environment variable IGNORE_FROM_REGEXP. It defaults to [^\w\W], so in effect ignoring nothing.
For every running container the docker log file is added and it is checked if a logstash-forwarder config exists within the container at
If an in container specific config exists, the path of all files will be expanded to be valid within the logstash-forwarder container before adding them to the global configuration.
This requires the following (in container defaults in brackets):
- read-only access to the directory containing your docker data (
- connection to Docker (
- connection to Logstash (
Read-only access to Docker data:
Mount the directory containing your Docker data into the containers
/var/lib/docker - i.e. run the container with
-v /var/lib/docker:/var/lib/docker:ro (assuming your Docker files are stored in
/var/lib/docker on the host).
Connection with Docker:
For communication with Docker the following endpoints are evaluated:
- whatever is passed via the
-dockercommand line flag
It is suggested to use the later - as in run the container with
Behind the screens fsouza/go-dockerclient is used for communication with Docker.
Connection with Logstash:
For communication with Logstash the following endpoints are evaluated:
- whatever is passed via the
-logstashcommand line flag
This allows you to
docker -link your Logstash instance to the containers
logstash-forwarder authentication can be managed in the following ways:
- specify a custom config pointing to some imported volume containing the required cert & key via the
-configflag (only the
networksection is evaluated)
- make your keys available bellow
TL;DR / Quickstart:
$ docker pull digitalwonderland/logstash-forwarder $ docker run -d --name logstash-forwarder -v /var/lib/docker:/var/lib/docker:ro -v /var/run/docker.sock:/var/run/docker.sock --link logstash:logstash --volumes-from logstash digitalwonderland/logstash-forwarder
If you start from scratch / use Vagrant / are on a Mac: just clone this repository and run
vagrant up. This gives you a VM based on CoreOS (which is awesome btw) running those 3 containers & Kibana listening to localhost:5601 (Docker listens to localhost:2375).
docker-logstash-forwarder must be run as root until Docker provides configurable ownership of shared volumes, because
/var/lib/dockeris owned by root on the host and mounted read only, so a non root user can not read from it (docker#7918).
The path of the containers content, on the hosts file system, has to be calculated by trying to take an educated guess based on your currently used docker driver since the docker folks consider this path internal and don't want to make it available via API (docker#7915).
Known to be working drivers are:
Last but not least it probably should be mentioned, that this is the first time I wrote any go code (a few days, after work), so any 'Duh' pointers are greatly appreciated.
Pull Requests welcome :)