Public | Automated Build

Last pushed: 18 days ago
Short Description
Performance Monitoring, Events and Logs for Docker, Docker Cloud, Docker UCP, Swarm, CoreOS, ...
Full Description

Image tags

  • latest - the latest release (stable)
  • X.Y.Z - release version (stable)
  • X.Y.Z-rc1-N - release canditates (unstable)
  • dev - build from github master branch (unstable)
  • swarm3k - custom build for SwarmZilla, used only for the swarm3k test.

Sematext Agent for Docker

Sematext Agent for Docker collects Metrics, Events and Logs from the Docker API for SPM Docker Monitoring & Logsene / Hosted ELK Log Management. Works with CoreOS, RancherOS, Docker Swarm, Kubernetes, Apache Mesos, Hashicorp Nomad, Amzon ECS, ... see installation.

How it works

Video: Log Management for Docker

Gathered information:

Type Description
Operating System Metrics Host machine metrics CPU / Mem / Swap / IO
Docker Container Metrics/Stats CPU Usage, Memory Usage, Network Stats, Disk I/O Stats
Events
Agent Startup Event server-info – created by spm-agent framework with node.js and OS version info on startup
docker-info – Docker Version, API Version, Kernel Version on startup
Docker Events
Container Lifecycle Events create, exec_create, destroy, export, ...
Container Runtime Events die, exec_start, kill, pause, restart, start, stop, unpause, ...
Docker Logs
default fields host / IP address, docker_host, container id, container name, image name, message
Log format detection and log parsers NGINX, Apache httpd, Kafka, Solr, HBase, Zookeeper, Cassandra, MySQL, MongoDB, Redis, Elasticsearch, Nsq.io
JSON, Plain Text

Installation

  1. Get a free account at sematext.com/spm
  2. Create an SPM App of type "Docker" and copy the SPM Application Token
  3. Run the image (please use your individual SPM and Logsene token!)

    docker pull sematext/sematext-agent-docker
    docker run -d --name sematext-agent-docker -e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN -v /var/run/docker.sock:/var/run/docker.sock sematext/sematext-agent-docker
    

    You’ll see your Docker metrics in SPM after about a minute.

  4. Watch metrics, use anomaly detection for alerts, create e-mail reports and much more ...

Docker Events:

Configuration Parameters

Parameter / Environment variable Description
Required Parameters
SPM_TOKEN SPM Application Token, enables metric and event collection
LOGSENE_TOKEN Logsene Application Token enables logging to Logsene, see logging specifc parameters for filter options and Log Routing section to route logs from different containers to separate Logsene applications
-v /var/run/docker.sock Path to the docker socket (optional, if dockerd provides TCP on 2375, see also DOCKER_PORT and DOCKER_HOST parameter)
TCP and TLS connection If the unix socket is not available Sematext Agent assumes the Container Gateway Address (autodetect) and port 2375 as default (no TLS) - this needs no configuration. In case the Docker Daemon TCP settings are different, you have to configure the TCP settings. The TCP settings can be modified with the following parameters
DOCKER_HOST e.g. tcp://ip-reachable-from-container:2375/ - default value 'unix:///var/run/docker.sock'. When the unix socket is not available the agent tries to connect to tcp://gateway:2375. In case a TCP socket is used there is no need to mount the Docker unix socket as volume
DOCKER_PORT Sematext Agent will use its gateway address (autodetect) with the given DOCKER_PORT
DOCKER_TLS_VERIFY 0 or 1
DOCKER_CERT_PATH Path to your certificate files, mount the path to the container with "-v $DOCKER_CERT_PATH:$DOCKER_CERT_PATH"
Configuration via docker swarm secrets:
CONFIG_FILE Path to the configuration file, containing environment variables key=value. Default value: /run/secrets/sematext-agent. Create a secret with docker secret create sematext-agent ./sematext-agent.cfg. Start Sematext Docker agent with `docker service create --mode global --secret sematext-agent --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock sematext/sematext-agent-docker
Optional Parameters:
--privileged The parameter might be helpful when Sematext Agent could not start because of limited permission to connect and write to the Docker socket /var/run/docker.sock. The privileged mode is a potential security risk, we recommend to enable the appropriate security. Please read about Docker security: https://docs.docker.com/engine/security/security/
HOSTNAME_LOOKUP_URL On Amazon ECS, a metadata query must be used to get the instance hostname (e.g. "169.254.169.254/latest/meta-data/local-hostname")
HTTPS_PROXY URL for a proxy server (behind firewalls)
LOGSENE_RECEIVER_URL URL for bulk inserts into Logsene. Required for Logsene On-Premises only.
SPM_RECEIVER_URL URL for bulk inserts into SPM. Required for SPM On-Premises only.
EVENTS_RECEIVER_URL URL for SPM events receiver. Required for SPM On-Premises only.
Docker Logs Parameters
TAGGING_LABELS A list of docker label names or environment variable names to tag container logs. Supporting wildcards e.g. TAGGING_LABELS='com.docker.swarm,com.myapp.'
Whitelist containers for logging
MATCH_BY_NAME Regular expression to white list container names
MATCH_BY_IMAGE Regular expression to white list image names
Blacklist containers
SKIP_BY_NAME Regular expression to black list container names
SKIP_BY_IMAGE Regular expression to black list image names for logging
PATTERNS_URL Load pattern.yml via HTTP e.g. -e PATTERNS_URL=https://raw.githubusercontent.com/sematext/logagent-js/master/patterns.yml
LOGAGENT_PATTERNS Pass patterns.yml via env. variable e.g. -e LOGAGENT_PATTERNS="$(cat ./patters.yml)"
PATTERN_MATCHING_ENABLED Activate logagent-js parser, default value is true. To disable the log parser set the value to false. This could increase the throughput of log processing for nodes with a very high log volume.
-v /yourpatterns/patterns.yml:/etc/logagent/patterns.yml to provide custom patterns for log parsing, see logagent-js
-v /tmp:/logsene-log-buffer Directory to store logs, in case of a network or service outage. Docker Agent deletes these files after successful transmission.
GEOIP_ENABLED trueenables GeoIP lookups in the log parser, default value: false
MAXMIND_DB_DIR Directory for the Geo-IP lite database, must end with /. Storing the DB in a volume could save downloads for updates after restarts. Using /tmp/ (ramdisk) could speed up Geo-IP lookups (consumes add. ~30 MB main memory).
ENABLE_LOGSENE_STATS Enables logging of tranmission stats to Logsene. Default value 'false'. Provides number of logs received, number of logs shipped, number of failed/successful http tranmissions (bulk requests to Logsene) and re-transmissions of failed requests.

Log Routing

Routing logs from different containers to separate Logsene Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN.
Sematext Agent inspects the containers for this Label and ships the logs to the defined Logsene App.

To disable logging to Logsene/Elasticsearch label the application container with LOGSENE_ENABLED=false. LOGSENE_ENABLED=true enables logging for the container again.

Example:
The following command will start nginx webserver and logs for this container will be shipped to the related Logsene App.

docker run --label LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGSENE_TOKEN -p 80:80 nginx
# or use environment variable on Kubernetes (no support for Docker labels)
# docker run -e LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGSENE_TOKEN -p 80:80 nginx

All other container logs will be shipped to the Logsene App specified in the docker run command for sematext/sematext-agent-docker with the environment variable LOGSENE_TOKEN.

Integrated Log Parser

SPM for Docker recognizes log formats - so your logs arrive in a structured format in Logsene!
The format recognition, data extractions, date parsing etc. is provided by logagent-js and covers:

  • Format detection e.g. for
    • Mongo DB
      • Nginx
      • Apache httpd, Kafka, Cassandra, HBase, Solr, Zookeeper
      • MySQL
      • Redis
  • plain text log messages
  • line delimited JSON logs
  • GeoIP enrichment for webserver logs, or any other field defined
    in the pattern definitions

To use a custom pattern definition simply mount a volume to '/etc/logagent/patterns.yml':

-v /mydir/patterns.yml:/etc/logagent/patterns.yml

Feel free to contribute to logagent-js to enrich the default pattern set.

Installation on CoreOS Linux

Sematext Agent for Docker can monitor CoreOS clusters including metrics and logs from Docker and journald.
See: Setup Sematext Agent on CoreOS

Installation on Kubernetes

Run Sematext Agent as Kubernetes DaemonSet.

  1. Get a free account at sematext.com/spm
  2. Create an SPM App of type "Docker" and copy the SPM Application Token
  3. Create sematext-agent.yml - and set your SPM and Logsene App Token in the section spec.env.
  4. Run the DaemonSet
kubectl create -f sematext-agent.yml

Installation on Docker Swarm

For Swarm on Docker engine > v1.12 use a global service to deploy the agent to all cluster nodes:

docker service create --mode global --reserve-memory 128mb --restart-condition any \
--name sematext-agent-docker \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
-e SPM_TOKEN=YOUR_SPM_TOKEN  \
-e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN \
sematext/sematext-agent-docker

Adjust the reserved memory to your needs (>70 MB).

Please read Docker Swarm: Collecting Metrics, Events & Logs for previous Swarm versions.

Installation on RancherOS

Please read RancherOS Monitoring and Logging Support there are various deployment options for Rancher, Swarm, Kubernetes or Mesos.

In addition we recommend to read Rancher Labs blog post about the RancherOS Catalog Entry.

Installation on Nomad by Hashicorp

See and example of the job description for Nomad by Hashicorp

Installation on Mesos / Marathon

Please note that you have to specify the number of Mesos nodes (instances), SPM App Token and Logsene App Token. Example call to the Marathon API:

curl -XPOST -H "Content-type: application/json" http://your_marathon_server:8080/v2/apps  -d '
{
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "sematext/sematext-agent-docker"
    },
    "volumes": [
      {
        "containerPath": "/var/run/docker.sock",
        "hostPath": "/var/run/docker.sock",
        "mode": "RW"
      }
    ],
    "network": "BRIDGE"
  },
  "env": {
        "LOGSENE_TOKEN": "YOUR_LOGSENE_TOKEN",
        "SPM_TOKEN": "YOUR_SPM_TOKEN" 
  },
  "id": "sematext-agent",
  "instances": 1,
  "cpus": 0.1,
  "mem": 100,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ]
}

Support

  1. Please check the SPM for Docker Wiki
  2. If you have questions about SPM for Docker, chat with us in the SPM user interface or drop an e-mail to support@sematext.com
  3. Open an issue here
  4. Contribution guide here
Docker Pull Command
Owner
sematext
Source Repository

Comments (1)
linhares
7 months ago

My environment:
Azure
CoreOs 1185.2
Rancher 1.1.4
Kubernetes

For this work on my environment I had to use
hostNetwork: true
inside spec of the daemon set