Stroom Log Sender

The purpose of this image is to make it easier for applications to send their log files to Stroom (or Stroom Proxy). It is a very small image that essentially runs cron to periodically send logs from one container into Stroom/Stroom Proxy.

Requirements for the log source

• The application generating the logs does not have to be running in a Docker container.
• The application should roll its log files (typically on a scheduled basis, e.g. 1min).
• The rolled log files should be in a directory that can be made available to Stroom Log Sender (with read/write access), e.g. inside a Docker managed volume if it is running in a container.

Configuring Stroom Log Sender

Stroom Log Sender can send logs from multiple applications at once. To set up Stroom Log Sender to send logs from an application do the following:

• If the application's rolled logs are in a docker volume, expose that volume to Stroom Log Sender with read/write access. These volumes should be mounted to the directory /stroom-log-sender/log-volumes/ inside the container.
• Use a bind mount to expose a host directory to /stroom-log-sender/config/ in Stroom Log Sender. This directory is used to hold the cron configuration file crontab.txt. This file must be present for the container to start and its contents will be loaded into the crontab each time the container starts up.
• Use a bind mount to expose a host directory to /stroom-log-sender/certs/ in Stroom Log Sender. This directory is used to hold the keys/certs required for sending the logs to Stroom/Proxy over two way SSL.

This is an example docker-compose configuration⁠ for Stroom Log Sender showing how the volumes are exposed to it. Typically one volume per log source is exposed.

This is an example crontab.txt⁠ that sends logs for ten log sources from three different containers.

The cron configuration uses the send_to_stroom.sh⁠ script to send the logs to stroom and delete them once successfully sent. The usage of this script is described by this Argbash⁠ spec file send_to_stroom_args.m4⁠

Minimum requirements to send to Stroom/Proxy

To send logs to Stroom or Stroom proxy you must provide the following values, which get added as HTTP header arguments. These values are used in Stroom to determine how the data should be stored and processed.

• Feed - A Feed defines a set of log files of the same format (e.g. apache logs) and typically for the same system, e.g. HR_SYSTEM-ACCESS-EVENTS.
• System - The name of the system that is producing the logs, e.g HR_SYSTEM.
• Environment - The type of environment e.g. OPS, DEV, etc.

Additional headers

If additional metadata needs to be sent to Stroom with the logs then additional HTTP headers can be used for this purpose. An example of this is where the log source is running on a different host to Stroom Log Sender, so the source system can write the details of its location as additional headers so stroom knows where the logs originally came from, e.g. OriginalHost:hr-system.some-domain.com.

All HTTP headers will be stored along side the log data by Stroom on receipt. The additional headers can be specified using the --headers argument to send_to_stroom.sh, for example /stroom-log-sender/log-volumes/hr-system/access/extra_headers.txt. Any headers found in this file will be sent with each log file. Headers Feed, System and Environment are specified as positional arguments to send_to_stroom.sh and therefore should not appear in this headers file.

For an example of how to set node/container specific values in a headers file see how the Stroom docker image does this:

• add_container_identity_headers.sh⁠
• Dockerfile⁠
• docker-entrypoint.sh⁠

An example of a headers file would be:

OriginalHost:server1.some.domain
OriginalIP:192.168.0.22
OriginalImageGitTag:v6.0-beta.30
OriginalContainerId:19c44f42e66e75de8e4c34e2d7079491588130424dd3817aec210e31289e99f5