gchq/stroom-log-sender
An image for the scheduled sending of log files to stroom
9.1K
The purpose of this image is to make it easier for applications to send their log files to Stroom (or Stroom Proxy). It is a very small image that essentially runs cron to periodically send logs from one container into Stroom/Stroom Proxy.
Stroom Log Sender can send logs from multiple applications at once. To set up Stroom Log Sender to send logs from an application do the following:
/stroom-log-sender/log-volumes/
inside the container./stroom-log-sender/config/
in Stroom Log Sender. This directory is used to hold the cron configuration file crontab.txt
. This file must be present for the container to start and its contents will be loaded into the crontab each time the container starts up./stroom-log-sender/certs/
in Stroom Log Sender. This directory is used to hold the keys/certs required for sending the logs to Stroom/Proxy over two way SSL.This is an example docker-compose configuration for Stroom Log Sender showing how the volumes are exposed to it. Typically one volume per log source is exposed.
This is an example crontab.txt that sends logs for ten log sources from three different containers.
The cron configuration uses the send_to_stroom.sh script to send the logs to stroom and delete them once successfully sent. The usage of this script is described by this Argbash spec file send_to_stroom_args.m4
To send logs to Stroom or Stroom proxy you must provide the following values, which get added as HTTP header arguments. These values are used in Stroom to determine how the data should be stored and processed.
HR_SYSTEM-ACCESS-EVENTS
.HR_SYSTEM
.OPS
, DEV
, etc.If additional metadata needs to be sent to Stroom with the logs then additional HTTP headers can be used for this purpose. An example of this is where the log source is running on a different host to Stroom Log Sender, so the source system can write the details of its location as additional headers so stroom knows where the logs originally came from, e.g. OriginalHost:hr-system.some-domain.com
.
All HTTP headers will be stored along side the log data by Stroom on receipt. The additional headers can be specified using the --headers
argument to send_to_stroom.sh
, for example /stroom-log-sender/log-volumes/hr-system/access/extra_headers.txt
. Any headers found in this file will be sent with each log file. Headers Feed
, System
and Environment
are specified as positional arguments to send_to_stroom.sh
and therefore should not appear in this headers file.
For an example of how to set node/container specific values in a headers file see how the Stroom docker image does this:
An example of a headers file would be:
OriginalHost:server1.some.domain
OriginalIP:192.168.0.22
OriginalImageGitTag:v6.0-beta.30
OriginalContainerId:19c44f42e66e75de8e4c34e2d7079491588130424dd3817aec210e31289e99f5
docker pull gchq/stroom-log-sender