robcowart/elastiflow-logstash-oss repository overview

⁠Running ElastiFlow™ on Docker

SUPPORTING ElastiFlow™ - Today literally 1000s of users leverage ElastiFlow™ As a powerful alternative to expensive commercial flow collecting solutions. As its popularity has increased, so has the time commitment necessary to support users and provide further enhancements. If you are one of the organizations who appreciate the value of ElastiFlow™, I would like to ask you to consider becoming a sponsor. The support from sponsors allows me dedicate more time and energy to the project. To become a sponsor, please visit ElastiFlow on .

⁠Setting up ElastiFlow™ on Docker

The easiest way to get ElastiFlow™ up and running quickly is to use Docker and docker-compose. The following instructions will walk you through setting up a single node installation of ElastiFlow on Docker.

NOTE: These instructions assume that you will have a server available with a recent Linux distribution and both Docker and docker-composer installed.

⁠Prepare the Data Path

Data written within a container's file system is ephemeral. It will be lost when the container is removed. For the data to persist it is necessary to write the data to local host's file system using a bind mount. You must create a path on the local host, and set the necessary permissions for the processes within the container to write to it.

sudo mkdir /var/lib/elastiflow_es
sudo chown -R 1000:1000 /var/lib/elastiflow_es

⁠Customize Environment Variables in docker-compose.yml

While the provided defaults should allow you to get up and running quickly, you may need to make changes specific to your requirements. After copying the provided docker-compose.yml from the repository to the server, edit any relevant environment variables.

The ElastiFlow™ Logstash container can be configured using the same environment variables discussed in INSTALL.md.

⁠Start the Elastic Stack using docker-compose

Start the Elastic Stack (incl. Logstash with the ElastiFlow pipeline) using docker-compose.

From the path where you placed the docker-compose.yml file run:

sudo docker-compose up -d

⁠Import Dashboards into Kibana

The Index Patterns, vizualizations and dashboards can be loaded into Kibana by importing the elastiflow.kibana.VERSION.json (https://github.com/robcowart/elastiflow/tree/master/kibana⁠) file from within the Kibana UI. This is done from the Management -- Saved Objects page.

You may also want to configure the recommend advanced Kibana settings discussed in INSTALL.md.

⁠docker-compose.yml

version: '3'

services:
  elastiflow-elasticsearch-oss:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.3.0
    container_name: elastiflow-elasticsearch-oss
    restart: 'no'
    ulimits:
      memlock:
        soft: -1
        hard: -1
    network_mode: host
    volumes:
      - /var/lib/elastiflow_es:/usr/share/elasticsearch/data
    environment:
      # JVM Heap size
      #   - this should be at least 2GB for simple testing, receiving only a few flows per second.
      #   - for production environments upto 31GB is recommended.
      ES_JAVA_OPTS: '-Xms2g -Xmx2g'

      cluster.name: elastiflow

      bootstrap.memory_lock: 'true'

      network.host: 0.0.0.0
      http.port: 9200
      discovery.type: 'single-node'

      indices.query.bool.max_clause_count: 8192
      search.max_buckets: 100000

      action.destructive_requires_name: 'true'

  elastiflow-kibana-oss:
    image: docker.elastic.co/kibana/kibana-oss:7.3.0
    container_name: elastiflow-kibana-oss
    restart: 'no'
    depends_on:
      - elastiflow-elasticsearch-oss
    network_mode: host
    environment:
      SERVER_HOST: 0.0.0.0
      SERVER_PORT: 5601

      ELASTICSEARCH_HOSTS: "http://127.0.0.1:9200"

      KIBANA_DEFAULTAPPID: "dashboard/653cf1e0-2fd2-11e7-99ed-49759aed30f5"

      LOGGING_QUIET: 'true'

  elastiflow-logstash-oss:
    image: robcowart/elastiflow-logstash-oss:3.5.1
    container_name: elastiflow-logstash-oss
    restart: 'no'
    depends_on:
      - elastiflow-elasticsearch-oss
    network_mode: host
    environment:
      # JVM Heap size - this MUST be at least 3GB (4GB preferred)
      LS_JAVA_OPTS: '-Xms4g -Xmx4g'

      # ElastiFlow global configuration
      ELASTIFLOW_DEFAULT_APPID_SRCTYPE: "__UNKNOWN"

      # Name resolution option
      ELASTIFLOW_RESOLVE_IP2HOST: "false"
      ELASTIFLOW_NAMESERVER: "127.0.0.1"

      ELASTIFLOW_NETFLOW_IPV4_PORT: 2055
      ELASTIFLOW_SFLOW_IPV4_PORT: 6343
      ELASTIFLOW_IPFIX_TCP_IPV4_PORT: 4739