The Gravwell File Follow ingester is designed to tail running log files and ship data to a Gravwell instance. Common systems like syslog, apache, and custom services typically log to rotating log files. The File follower system can detect new log files, monitor new data, and let go of rotated files.
The File Follower Docker Image is also an excellent choice for sidecar deployments where component is used to synchronize state across multiple Gravwell webservers when running in a distributed frontend mode.
A single File Follower daemon can monitor multiple directories at once.
For full configuration details, see the FileFollower Documentation.
The File Follower component uses a configuration file located at /opt/gravwell/etc/file_follow.conf which controls the directories monitored and the upstream ingest configuration.
The following is an minimal example of a file_follow.conf:
[Global]
#Ingest-Secret = IngestSecrets
Connection-Timeout = 0
Insecure-Skip-TLS-Verify=false
##Cleartext-Backend-Target=127.0.0.1:4023 #example of adding a cleartext connection
##Cleartext-Backend-Target=127.1.0.1:4023 #example of adding another cleartext connection
##Encrypted-Backend-Target=127.1.1.1:4024 #example of adding an encrypted connection
#Pipe-Backend-Target=/opt/gravwell/comms/pipe #a named pipe connection, this should be used when ingester is on the same machine as a backend
State-Store-Location=/opt/gravwell/etc/file_follow.state
Log-Level=INFO #options are OFF INFO WARN ERROR
Log-File=/opt/gravwell/log/file_follow.log
#Ingest-Cache-Path=/opt/gravwell/cache/file_follow.cache # because we're usually dealing with files on disk, we disable the ingest cache by default
#Max-Ingest-Cache=1024 #Number of MB to store, localcache will only store 1GB before stopping. This is a safety net
Max-Files-Watched=64 # Maximum number of files to watch before rotating out old ones, this can be bumped but will need sysctl flags adjusted
#basic default logger, all entries will go to the default tag
#no Tag-Name means use the default tag
[Follower "auth"]
Base-Directory="/var/log/"
File-Filter="auth.log,auth.log.[0-9]" #we are looking for all authorization log files
Tag-Name=auth
Assume-Local-Timezone=true #Default for assume localtime is false
[Follower "packages"]
Base-Directory="/var/log"
File-Filter="dpkg.log,dpkg.log.[0-9]" #we are looking for all dpkg files
Tag-Name=dpkg
Ignore-Timestamps=true
The Docker deployment enables some feature configuration and control using environment variables. Each Gravwell docker component ships with the Crash Reporting Service which can be disabled using environment variables.
Add DISABLE_ERROR_HANDLER=TRUE as an environment variable to disable the crash reporter.
Free community licenses for Gravwell are available.
Content type
Image
Digest
sha256:e9f14d85a…
Size
21.5 MB
Last updated
5 days ago
docker pull gravwell/file_follow