Public | Automated Build

Last pushed: 2 months ago
Short Description
Full Description


The searchlog is based on the (R)ELK stack - Redis, Elasticsearch, Logstash and Kibana. For logshipping we use Filebeat (by Elastic) and Redis has been added to the mix as a buffer/broker (in order to handle event spikes).

#####Shipping Logdata
When shipping logs, the actual flow can be described as follows:

FileBeat (logshipper) >> Redis << Logstash (indexer) >> Elasticsearch << Kibana

Filebeat ships aggregated logdata to a Redis queue. Logstash pulls messages from this queue, and messages (logdata) are then parsed and processed by Logstash which, eventually, are then outputting this data into Elasticsearch to be stored. Finally Kibana is used as a data visualization platform, allowing us to interact with our data.


  1. Setup Docker Environment
  2. Setup Filebeat

Setup Environment

1.1 | Docker-Toolbox

  • Download and install Docker-Toolbox with virtualbox (or Docker for Windows).
    (current version is: DockerToolbox-1.11.2)
    Install using the default options (assuming git is already installed)

  • Start Docker Quickstart Terminal. The first time you start the terminal, a new (VirtualBox) Virtual Machine named "default" will created during startup.

  • Verify that docker is working - ie. by listing avaiable docker-machines...

$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   *        virtualbox   Running   tcp://           v1.11.2
  • If this is your first time using docker, you might wanna run your fist docker container using the "docker/whalesay" image - just to start out gently :)
$ docker run docker/whalesay cowsay boo
Unable to find image 'docker/whalesay:latest' locally
latest: Pulling from docker/whalesay

e190868d63f8: Pull complete ================================================>] 65.77 MB/65.77 MBB
909cd34c6fd7: Pull complete ================================================>] 71.48 kB/71.48 kBB
0b9bfabab7c1: Pull complete ================================================>]    682 B/682 BB
a3ed95caeb02: Pull complete ================================================>]     32 B/32 BB
00bf65475aba: Pull complete ================================================>] 37.71 MB/37.71 MBB
c57b6bcc83e3: Pull complete ================================================>] 61.68 kB/61.68 kBB
8978f6879e2f: Pull complete ================================================>] 18.32 kB/18.32 kBB
8eed3712d2cf: Pull complete ================================================>] 11.62 kB/11.62 kBB
Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
Status: Downloaded newer image for docker/whalesay:latest
< boo >
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/

1.2 | Build Docker-Compose

Assuming everything is working, it's time to checkout our docker-compose script.
Navigate to the docker-compose files located in the Gyldendal.Logging/Environments/DEV directory, and build the compose file.

$ Miracle@DESKTOP ~/Source/Gyldendal.Logging/Environments/Dev (master)
$ docker-compose build

This will build docker-images for each of our custom dockerfiles referenced in our compose-file.yml. The first time you build a dockerfile, it will take quite a while.

When the build finishes, verify that all docker images was build sucessfully - using the "docker images" command:

Miracle@DESKTOP ~/Source/Gyldendal.Logging/Environments/Dev (master)
$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
dev_kibana           latest              159341bc0d3c        2 minutes ago       324.9 MB
dev_logstash         latest              af09b4353645        4 minutes ago       457 MB
dev_elastic_data     latest              4f2f991305bc        6 minutes ago       386.7 MB
dev_elastic_master   latest              4f2f991305bc        6 minutes ago       386.7 MB
ubuntu               trusty              b2f1fdd93175        3 days ago          188.4 MB
java                 8-jre               76fd51ceaa2e        6 weeks ago         312.2 MB
docker/whalesay      latest              6b362a9f73eb        14 months ago       247 MB

Next, say your prayers and take it for a spin :)

Miracle@DESKTOP ~/Source/Gyldendal.Logging/Environments/Dev (master)
$ docker-compose up

That's it - we're done setting up our docker environment.


Unable to mount volumes

In case your code isn't located within your HOME (~/User) directory (your codebase might be located in C:/Soruce like mine), chances are you'll get an IllegalStateException similar to the one below...

$ docker-compose up 
Creating dev_elastic_master_1
Creating REDIS
Creating dev_kibana_1
Creating dev_elastic_data_1
elastic_master_1  | log4j:WARN No appenders could be found for logger (bootstrap).
elastic_master_1  | log4j:WARN Please initialize the log4j system properly.
elastic_master_1  | log4j:WARN See for more info.
elastic_master_1  | Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.scripts' (/usr/share/elasticsearch/config/scripts) <<--- 
elastic_master_1  | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/scripts
elastic_master_1  |     at sun.nio.fs.UnixException.translateToIOException(
elastic_master_1  |     at sun.nio.fs.UnixException.rethrowAsIOException(
elastic_master_1  |     at sun.nio.fs.UnixException.rethrowAsIOException(

This might be caused by insufficient permissions. To fix this, you'll need to mount a new shared directory on your virtualbox machine, mapped to whatever folder where your codebase is located - otherwise your VM wont be able to mount volumes from your HOST.

How to mount a custom shared folder as a VirtualBox volume.
Fx. If our codebase is located in the C:/Source directory, we would have to add a new shared directory named "Source" to our VBox machine and mount it manually.

docker-machine ssh default "sudo mkdir -p /c/Source"
docker-machine ssh default "sudo mount -t vboxsf Source /c/Source"
sudo sysctl -w net.core.somaxconn=65535
sudo sysctl vm.overcommit_memory=1
sudo echo never > /sys/kernel/mm/transparent_hugepage/enabled

Running the Redis server

  1. Raise somaxconn above 511

1417:M 25 Oct 06:13:31.840 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
To fix this warning you have to set a new config to /etc/rc.local so that the setting will persist upon reboot

$~: sudo nano /etc/rc.local

Add this:

sysctl -w net.core.somaxconn=65535

...or just echo it directyly to the file insted:

echo 'sysctl -w net.core.somaxconn=65535' >> /etc/rc.local

When you reboot the next time, the new setting will be to allow 65535 connections instead of 128 as before.

2) vm.overcommit_memory = 1

Redis highly recommends enabling VM memory overcommit. Docker cannot do this on a container-only basis, it must be enabled on your Docker host system. To immediately enable it:

$ docker-machine.exe ssh default "sudo sysctl vm.overcommit_memory=1"

If the value is not set, or is set to 0, then this container will still run but background saves by Redis might fail. If you have already started docker-redis with overcommit disabled, you can stop the container with docker, set the overcommit setting on your docker host, and then start your docker-redis container again.

Alternayively, simply echo the needed line into the correct file:

echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf

You can check that it was added with this command

cat /etc/sysctl.conf
3) Disable THP

This is also simple to fix by just running the recommended command as stated in the warning.

echo never > /sys/kernel/mm/transparent_hugepage/enabled

...and make sure it'' be persitent whenever we reboot:

echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

You can check that it was added with this command
Add this:

cat /etc/rc.local

Now this will be persistent upon reboot as well.

Setup Filebeat

2.1 | Download and install Filebeat

The searchlog is based on the (R)ELK stack - Redis, Elasticsearch, Logstash and Kibana. For logshipping we use Filebeat (by Elastic) and Redis has been added to the mix as a buffer/broker (in order to handle event spikes).

Filebeat LogShipper

Filebeat is a shipper for log-data initially based on the Logstash-Forwarder. Installed as an agent on our servers, Filebeat monitors the log directories or specific log files, tails the files, and forwards them to whatever output we've defined (Redis).

A few words on how Filebeat works: When you start Filebeat, it starts one or more prospectors that look in the paths you’ve specified for log files. For each log file that the prospector locates, Filebeat starts a harvester. Each harvester reads a single log file for new content and sends the new log data to the spooler, which aggregates the events and sends the aggregated data to the output that you’ve configured for Filebeat. In our case we use the Redis output (acting as a broker) plugin, which is compatible with the Redis input plugin for Logstash.

2.2 | Configuring Filebeat

Filebeat is configured using a filebeat.yml-file, and in our specific case, we've defined two prospectors - one for monitoring searchlogger logfiles, and one for monitoring connectorclient logfiles.

   - "C:/Goo.Logs/Api/*.searchlogger.*.json"    
  document_type: searchlogger
   - "C:/Goo.Logs/Api/*.connectorclient.*.json" 
  document_type: connectorclient

Additionally we have defined two outputs used for data collected by the beat - a redis and a file output. The latter being primarily for debugging. As of July 2016, supported outputs are console, elasticsearch, file, logstash and redis.

   # Path to the directory where to save the generated files. The option is mandatory.
   path: "/tmp/filebeat"      
   filename: filebeat.gyldendal.json

    # Set the host and port where to find Redis.
    host: ""
    port: 6379

Essentially, our redis queue will receive data taged with a ‘type’ (specified in document_type). This type tells logstash later on, HOW to parse and process that log - i.e. which filters to apply.

2.3 | Running Filebeat

Recepie for shipping logs using Filebeat:

  1. Navigate to the "Gyldendal.Logging/Filebeat " directory and configure prospectors and outputs.
  2. You have to options...
    1. Run it manually, using the command "./filebeat.exe -c ./filebeat.yml"
    2. Install it as a service using the provided "install-service-filebeat.ps1" powershell script.
Miracle@DESKTO ~/Source/Gyldendal.Logging/Filebeat (master)
$ ls
total 10150
-rwxr-xr-x 1 Miracle None 10361856 May 17 20:33 filebeat.exe
-rw-r--r-- 1 Miracle None      814 May 17 20:33 filebeat.template.json
-rw-r--r-- 1 Miracle None    21891 Jul 22 12:31 filebeat.yml
-rw-r--r-- 1 Miracle None      442 May 17 20:33 install-service-filebeat.ps1
-rw-r--r-- 1 Miracle None      184 May 17 20:33 uninstall-service-filebeat.ps1
Miracle@DESKTOP ~/Source/Gyldendal.Logging/Filebeat (master)
$ ./filebeat.exe -c ./filebeat.yml -e -d "publish"

By default, Filebeat sends all its output to syslog. When you run Filebeat in the foreground, you can use the "-e" command line flag to redirect the output to standard error instead. In the sample above, we've increased the verbosity of debug messages by enabling a debug selector to view the "published" transactions.

How to re-import logs?

If you want to re-import/ship all logs from scratch, you'll need to delete the Filebeat registry-file.
By default the registry-file is located located in the "C:/ProgramData/filebeat" directory. However, this setting can be configured to suite your needs, under the general filebeat configuration options:


registry_file: "C:/ProgramData/filebeat/ordbog.registry"

To reship your logfiles, all you have to do, is delete the registry-file, restart filebeat,...and off they go once again.

Docker Pull Command
Source Repository

Comments (0)