Sign inSign up

fengzhou/logspout

By fengzhou

•Updated about 9 years ago

Collect docker logs from host containers, and send to Kafka, syslog, httpstream, or Logstash.

Image
1

365

fengzhou/logspout repository overview

⁠Change from github fork upstream

This docker contains logspout for kafka, syslog, httpstream, logstash.

For kafka, the special Info fields are exposed to simplify get all docker information, including host ip.

⁠KAFAK_TEMPLATE

Following example shows kafka template and using graylog format:

# Content in logspout.env via docker run --env-file option
KAFKA_TEMPLATE={"version": "1.1", "host": "{{.Info.HostName}}", "short_message": {{.Data | json}}, "timestamp": "{{.Time | timestamp }}", "level": {{.Level}}, "_container_id": "{{.Info.ContainerId}}", "_container_name": "{{.Info.ContainerName}}", "_message_source": "{{.Source}}", "_image_id": "{{.Info.ImageId}}", "_image_name": "{{.Info.ImageName}}", "_created": "{{.Container.Created | timestamp }}", "_host_id": "{{.Info.HostId}}", "_host_addr": "{{.Info.HostAddr}}", "_host_ip": "{{.Info.HostIp}}", "_command": {{.Info.Command | json }}, "_stack": "{{.Info.Stack}}", "_service": "{{.Info.Service}}" }

To support graylog level field, the field .Level is used.

  • Standard Output: LOG_INFO (6)
  • Standard Error: LOG_ERR (3)
  • Others: LOG_NOTICE (5)

⁠Notes for docker swarm service

Since go template is also used for docker swarm service to evaluate environments, use escape if function (like json, timestamp) is used.

# Content in logspout.env via docker service --env-file option
KAFKA_TEMPLATE={{`{"version": "1.1", "host": "{{.Info.HostName}}", "_message_source": "{{.Source}}", "short_message": {{.Data | json}}, "timestamp": "{{.Time | timestamp }}", "level": {{.Level}}, "_container_id": "{{.Info.ContainerId}}", "_container_name": "{{.Info.ContainerName}}", "_container_hostname": "{{.Container.Config.Hostname}}", "_container_created": "{{.Container.Created | timestamp }}", "_container_command": {{.Info.Command | json }}, "_container_image_id": "{{.Info.ImageId}}", "_container_image_name": "{{.Info.ImageName}}", "_container_stack": "{{.Info.Stack}}", "_container_service": "{{.Info.Service}}" }`}}

⁠Build

docker build -t <tag> .

⁠Usage (Based on github fork upstream)

docker run --rm --name="logspout" \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    -e KAFKA_TEMPLATE="{\"time\":\"{{.Time | timestamp }}\",\"container_name\":\"{{.Info.ContainerName}}\",\"source\":\"{{.Source}}\",\"data\":{{.Data | json }}, \"origin\": \"{{.Data}}\" }" \
    fengzhou/logspout \
    kafka://kafkahost:9092?topic=container-logs

# multi kafka
docker run --rm --name="logspout" \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    -e KAFKA_TEMPLATE="{\"time\":\"{{.Time | timestamp }}\",\"container_name\":\"{{.Info.ContainerName}}\",\"source\":\"{{.Source}}\",\"data\":{{.Data | json }}, \"origin\": \"{{.Data}}\" }" \
    fengzhou/logspout \
    kafka://broker1:9092?topic=container-logs&brokers=broker2:9092|broker3:9092

# enable ssl
# you need three certs in local directory and add ssl=true parameter
# eg.:
# /tmp/certs/kafka/
# ├── ca.pem
# ├── client_cert.pem
# └── client_key.pem
docker run --rm --name="logspout" \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    --volume=/tmp/certs/kafka:/data/certs/kafka \
    -e KAFKA_TEMPLATE="{\"time\":\"{{.Time | timestamp }}\",\"container_name\":\"{{.Info.ContainerName}}\",\"source\":\"{{.Source}}\",\"data\":{{.Data | json }}, \"origin\": \"{{.Data}}\" }" \
    fengzhou/logspout \
    kafka://broker1:9092?topic=container-logs&brokers=broker2:9092|broker3:9092&ssl=true

Tag summary

Content type

Image

Digest

Size

14.1 MB

Last updated

about 9 years ago

docker pull fengzhou/logspout