pixelfederation/mesos-slave repository overview

⁠- Mesos Slave -

An Ubuntu based Mesos Slave container, packaged with Logstash-Forwarder and managed by Supervisord. All parameters are controlled through environment variables, with some settings auto-configured based on the environment.

⁠Version Information:

• Container Release: 1.2.1
• Mesos: 0.28.2-2.0.27.ubuntu1404
• Docker: 1.11.2-0~trusty

Services Include:

• Mesos Slave⁠ - Primary process that offers resources of the host to the Mesos Master(s) for scheduling and running of tasks.
• Consul-Template⁠ - An application that can populate configs from a consul service.
• Logrotate⁠ - A script and application that aid in pruning log files.
• Logstash-Forwarder⁠ - A lightweight log collector and shipper for use with Logstash⁠.
• Redpill⁠ - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
• Rsyslog⁠ - The system logging daemon.

⁠Index

• Usage⁠
• Example Run Command⁠
• Modification and Anatomy of the Project⁠
• Important Environment Variables⁠
• Service Configuration⁠
• Mesos⁠
• Consul-Template⁠
• Logrotate⁠
• Logstash-Forwarder⁠
• Redpill⁠
• Rsyslog⁠
• Troubleshooting⁠

⁠Usage

All mesos commands should be passed via environment variables (please see the example run command⁠ below). For Mesos documentation, please see the configuration docs associated with the release here: mesos@d3717e5⁠

In a local proof of concept environment, the only variable that MUST be defined is MESOS_MASTER.

However, that will leave the slave with a fraction of it's functionality. To run in a useful fashion, the following should be set, ENVIRONMENT, MESOS_MASTER, MESOS_WORK_DIR, and MESOS_DOCKER_SANDBOX_DIRECTORY.

• ENVIRONMENT - when set to production or development it will enable all services including: mesos-master, logstash-forwarder, and redpill.

• MESOS_MASTER- Informs the slave how to connect or discover the Mesos Masters. Please see the Mesos docs for the available options.

• MESOS_WORK_DIR - Path to the directory in which framework directories are placed.

• MESOS_SANDBOX_DIRECTORY - Path to directory used to map the sandbox to Docker containers.

In addition to the above, there are several things to note for full compatibility when operating a mesos slave in a container with docker as a supported containerizer.

• The container should be run with host networking.
• The container requires several volumes to be mounted. For compatibility purposes these should be the same location as it is on the host.
• /usr/bin/docker:/usr/bin/docker:ro
• /var/run/docker.sock:/var/run/docker.sock:rw
• /sys:/sys:ro
• The directory used for MESOS_WORK_DIR as rw
• The directory used for MESOS_SANDBOX_DIRECTORY as rw

⁠Marathon Framework and Private Registry Access

Configuring private registry access is dependant on several factors. For documentation, please visit the Marathon Framework site directly⁠.

In either case, if you intend on baking the credentials into the image. This would be the image to do it.

⁠Example Run Command

docker run -d --net=host    \
--pid=host                  \
--name=mesosslave           \
--cap-add=SYS_ADMIN         \
-e ENVIRONMENT=production   \
-e PARENT_HOST=$(hostname)  \
-e MESOS_IP=10.10.0.111     \
-e MESOS_MASTER=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos  \
-e MESOS_REGISTRATION_TIMEOUT=5min    \
-e MESOS_CONTAINERIZERS=docker,mesos  \
-e MESOS_HOSTNAME=10.10.0.111         \
-e MESOS_SANDBOX_DIRECTORY=/data/mesos/sandbox   \
-e MESOS_WORKDIR=/data/mesos/workdir             \
-v /data/mesos/workdir:/data/mesos/workdir:rw    \
-v /data/mesos/sandbox:/data/mesos/sandbox:rw    \
-v /usr/bin/docker:/usr/bin/docker:ro            \
-v /var/run/docker.sock:/var/run/docker.sock:rw  \
-v /sys/:/sys:ro  \
mesos-slave

⁠Modification and Anatomy of the Project

File Structure The directory skel in the project root maps to the root of the file system once the container is built. Files and folders placed there will map to their corresponding location within the container.

Init The init script (./init.sh) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.

Supervisord All supervisord configs can be found in /etc/supervisor/conf.d/. Services by default will redirect their stdout to /dev/fd/1 and stderr to /dev/fd/2 allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.

In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.

Logstash-Forwarder The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERVICE_LOGSTASH_FORWARDER_ADDRESS environment variable.

In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.

⁠Important Environment Variables

⁠Defaults

⁠Description

• ENVIRONMENT_INIT - If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables.

• APP_NAME - A brief description of the container. If Logstash-Forwarder is enabled, this will populate the app_name field in the Logstash-Forwarder configuration file.

• ENVIRONMENT - Sets defaults for several other variables based on the current running environment. Please see the environment⁠ section for further information. If logstash-forwarder is enabled, this value will populate the environment field in the logstash-forwarder configuration file.

• PARENT_HOST - The name of the parent host. If Logstash-Forwarder is enabled, this will populate the parent_host field in the Logstash-Forwarder configuration file.

• MESOS_LOG_DIR - The path to the directory in which Mesos stores its logs.

• MESOS_WORK_DIR - Path to the directory in which framework directories are placed.

• GLOG_max_file_size - The size in Megabytes that the mesos log file(s) will be allowed to grow to before rotation.

• SERVICE_CONSUL_TEMPLATE - * SERVICE_CONSUL_TEMPLATE- Enables or disables the consul-template service. If enabled, it will also enableSERVICE_LOGROTATEandSERVICE_RSYSLOGto handle logging. (**Options:**enabledordisabled`)

• SERVICE_LOGROTATE - Enables or disabled the Logrotate service. This will be set automatically depending on the environment. (Options: enabled or disabled)

• SERVICE_LOGROTATE_INTERVAL - The time in seconds between runs of logrotate or the logrotate script. The default (3600 or 1 hour) is set by default in the logrotate script automatically.

• SERVICE_LOGROTATE_SCRIPT - The path to the script that should be executed instead of logrotate itself to clean up logs.

• SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

• SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

• SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

• SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

• SERVICE_RSYSLOG - Enables of disables the rsyslog service. This is managed by SERVICE_CONSUL_TEMPLATE, but can be enabled/disabled manually. (Options: enabled or disabled)

Environment

• local (default)

• prod|production|dev|development

• debug

* Only set if SERVICE_CONSUL_TEMPLATE is set to enabled.

⁠Service Configurations

⁠Mesos-Slave

As stated in the Usage⁠ section, Mesos-slave configuration information can be found in the github docs releated to the Mesos Release: mesos@d3717e5⁠.

The actual mesos start command is passed to supervisor via the SERVICE_MESOS_CMD environment variable, and defaults to mesos-slave.

⁠Mesos-Slave Environment Variables

⁠Defaults

⁠Description

• MESOS_CONTAINERIZES - Comma separated list of containerizers for use with Mesos. Priority is assigned in the order in which they're passed.

• MESOS_LOG_DIR - The path to the directory in which Mesos stores its logs.

• MESOS_WORK_DIR - Path to the directory in which framework directories are placed.

• SERVICE_MESOS_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Consul-Template

Provides initial configuration of consul-template. Variables prefixed with CONSUL_TEMPLATE_ will automatically be passed to the consul-template service at runtime, e.g. CONSUL_TEMPLATE_SSL_CA_CERT=/etc/consul/certs/ca.crt becomes -ssl-ca-cert="/etc/consul/certs/ca.crt". If managing the application configuration is handled via file configs, no other variables must be passed at runtime.

⁠Consul-Template Environment Variables

⁠Defaults

⁠Logrotate

The logrotate script is a small simple script that will either call and execute logrotate on a given interval; or execute a supplied script. This is useful for applications that do not perform their own log cleanup.

⁠Logrotate Environment Variables

⁠Defaults

⁠Description

• SERVICE_LOGROTATE - Enables or disables the Logrotate service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

• SERVICE_LOGROTATE_INTERVAL - The time in seconds between run of either the logrotate command or the provided logrotate script. Default is set to 3600 or 1 hour in the script itself.

• SERVICE_LOGROTATE_CONFIG - The path to the logrotate config file. If neither config or script is provided, it will default to /etc/logrotate.conf.

• SERVICE_LOGROTATE_SCRIPT - A script that should be executed on the provided interval. Useful to do cleanup of logs for applications that already handle rotation, or if additional processing is required.

• SERVICE_LOGROTATE_FORCE - If present, passes the 'force' command to logrotate. Will be ignored if a script is provided.

• SERVICE_LOGROTATE_VERBOSE - If present, passes the 'verbose' command to logrotate. Will be ignored if a script is provided.

• SERVICE_LOGROTATE_DEBUG - If present, passed the 'debug' command to logrotate. Will be ignored if a script is provided.

• SERVICE_LOGROTATE_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Logrotate Script Help Text

root@ec58ca7459cb:/opt/scripts# ./logrotate.sh --help
logrotate.sh - Small wrapper script for logrotate.
-i | --interval     The interval in seconds that logrotate should run.
-c | --config       Path to the logrotate config.
-s | --script       A script to be executed in place of logrotate.
-f | --force        Forces log rotation.
-v | --verbose      Display verbose output.
-d | --debug        Enable debugging, and implies verbose output. No state file changes.
-h | --help         This usage text.

⁠Supplied Cleanup Script

The below cleanup script will remove all but the latest 5 rotated logs.

#!/bin/bash

mld=${MESOS_LOG_DIR:-/var/log/mesos}

cd "$mld"

(ls -t | grep 'log.INFO.*'|head -n 5;ls)|sort|uniq -u|grep 'log.INFO.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.ERROR.*'|head -n 5;ls)|sort|uniq -u|grep 'log.ERROR.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.WARNING.*'|head -n 5;ls)|sort|uniq -u|grep 'log.WARNING.*'|xargs --no-run-if-empty rm

#consul-template uses rsyslog for logging, need to run logrotate to handle that log
if [[ "$SERVICE_CONSUL_TEMPLATE" == "enabled" ]]; then
  /usr/sbin/logrotate /etc/logrotate.conf
fi

⁠Logstash-Forwarder

Logstash-Forwarder is a lightweight application that collects and forwards logs to a logstash server endpoint for further processing. For more information see the Logstash-Forwarder⁠ project.

⁠Logstash-Forwarder Environment Variables

⁠Defaults

⁠Description

• SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

• SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

• SERVICE_LOGSTASH_FORWARDER_ADDRESS - The address of the Logstash server.

• SERVICE_LOGSTASH_FORWARDER_CERT - The path to the Logstash-Forwarder server certificate.

• SERVICE_LOGSTASH_FORWARDER_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Rsyslog

Rsyslog is a high performance log processing daemon. For any modifications to the config, it is best to edit the rsyslog configs directly (/etc/rsyslog.conf and /etc/rsyslog.d/*).

⁠Defaults

⁠Description

• SERVICE_RSYSLOG - Enables or disables the rsyslog service. This will automatically be set depending on what other services are enabled. (Options: enabled or disabled)

• SERVICE_RSYSLOG_CONF - The path to the rsyslog configuration file.

• SERVICE_RSYSLOG_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Redpill

Redpill is a small script that performs status checks on services managed through supervisor. In the event of a failed service (FATAL) Redpill optionally runs a cleanup script and then terminates the parent supervisor process.

⁠Redpill Environment Variables

⁠Defaults

⁠Description

• SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

• SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

• SERVICE_REDPILL_INTERVAL - The interval in which Redpill polls supervisor for status checks. (Default for the script is 30 seconds)

• SERVICE_REDPILL_CLEANUP - The path to the script that will be executed upon container termination.

• SERVICE_REDPILL_CMD - The command that is passed to supervisor. It is dynamically built from the other redpill variables. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Redpill Script Help Text

root@c90c98ae31e1:/# /opt/scripts/redpill.sh --help
Redpill - Supervisor status monitor. Terminates the supervisor process if any specified service enters a FATAL state.

-c | --cleanup    Optional path to cleanup script that should be executed upon exit.
-h | --help       This help text.
-i | --interval   Optional interval at which the service check is performed in seconds. (Default: 30)
-s | --service    A comma delimited list of the supervisor service names that should be monitored.

⁠Troubleshooting

In the event of an issue, the ENVIRONMENT variable can be set to debug. This will stop the container from shipping logs and prevent it from terminating if one of the services enters a failed state.

For mesos itself, the MESOS_LOGGING_LEVEL variable can be set to INFO or WARNING to further diagnose the problem.

If further information is needed; logging may be controlled directly by configuring glog⁠ loggig library used by Mesos. For reference; please see the table below:

Note: not all of the glog settings are overridable. Mesos does dictate some of them. Main ones of note are GLOG_v to increase log verbosity, and GLOG_max_log_size if log growth and rotation must be tuned.