lucamora/fluentd-humio repository overview

⁠Overview

This Docker image⁠ based on fluentd⁠ image is able to send Docker logs to Humio⁠.

⁠Outgoing log format

{
	"message": "<CONTAINER_LOG>",
	"@service": "<SERVICE_NAME>",
	"@type": "<DOCKER_PARSER>",
	"@container": "<YOUR_CONTAINER_ID>",
	"@shipper": "<SHIPPER_CONTAINER_ID>"
}

Where:

• <CONTAINER_LOG>: is the log generated by your app, read by Docker daemon and sent to the shipper in the log field
• <SERVICE_NAME>: is the name of your application configured on container creation (see setup container⁠)
• <DOCKER_PARSER>: is the parser that Humio use to parse the log (see setup shipper⁠)
• <YOUR_CONTAINER_ID>: is the ID of your app's container generated by the Docker daemon
• <SHIPPER_CONTAINER_ID>: is the ID of the shipper container generated by the Docker daemon

⁠Setup shipper container

To run the shipper container, two environment variables must be set:

• HUMIO_TOKEN: Humio ingest token
• DOCKER_PARSER: parser used in Humio to parse logs generated by Docker container

The easiest method to run the shipper is to create a standalone Docker container.

⁠Docker command

docker run \
-d \
-p 24224:24224 \
-e HUMIO_TOKEN=<INGEST_TOKEN> \
-e DOCKER_PARSER=<PARSER> \
--name shipper \
lucamora/fluentd-humio:2.0.0

Another way to run the shipper is to use the compose file (suggested method):

This example assumes that a swarm has already been initialized

⁠docker-compose.yml file

version: "3"
services:
  shipper:
    image: lucamora/fluentd-humio:2.0.0
    deploy:
      mode: global
      restart_policy:
        condition: any
    ports:
      - "24224:24224"
    environment:
      HUMIO_TOKEN: <INGEST_TOKEN>
      DOCKER_PARSER: <PARSER>

⁠Docker command

docker stack deploy -c docker-compose.yml humio

⁠Setup Docker container

To enable log shipping for a container some configuration has to be setup. First the --log-driver option must be set to fluentd, then one logging option has to be configured with --log-opt to identify the log source inside the Humio dashboard.

The easiest method to configure the container is to run it in standalone mode.

⁠Docker command

docker run \
--log-driver=fluentd \
--log-opt tag="docker.<SERVICE_NAME>.{{.ID}}" \
<YOUR_IMAGE>

Another way to configure the container is to use the compose file (suggested method):

This example assumes that a swarm has already been initialized

⁠docker-compose.yml file

version: "3"
services:
  web:
    image: <YOUR_IMAGE>
    logging:
      driver: fluentd
      options:
        tag: docker.<SERVICE_NAME>.{{.ID}}

⁠Docker command

docker stack deploy -c docker-compose.yml <STACK_NAME>

⁠Custom configuration

If you want to edit the configuration file without building your own image you can edit the original file⁠ and then bind it into the Docker container using the -v option.

If you run your app in standalone mode:

⁠Docker command

docker run \
... \
-v "path/to/your/fluent.conf:/fluentd/etc/" \
lucamora/fluentd-humio:2.0.0

Or if you use the docker-compose file

⁠docker-compose.yml file

version: "3"
services:
  shipper:
    image: lucamora/fluentd-humio:2.0.0
    ...
    volumes:
      - "path/to/your/fluent.conf:/fluentd/etc/"

⁠Migration from v1.1.0 to v2.0.0

The previous version of the shipper worked the same of the current version, but some details in the configuration have changed. The only difference from v.1.1.0 is that the <SERVICE_NAME> is shipped with @service field instead of service.

⁠Shipper container

In the configuration rename HUMIO_PARSER with DOCKER_PARSER.

⁠App container

Remove the service label from the logging options and update the container tag with docker.<SERVICE_NAME>.{{.ID}}.