pixelfederation/mesos-master repository overview

⁠- Mesos Master -

An Ubuntu based Mesos Master container, packaged with Logstash-Forwarder and managed by Supervisord. All parameters are controlled through environment variables, with some settings auto-configured based on the environment.

⁠Version Information:

• Container Release: 1.3.0
• Mesos: 1.1.0-2.0.107.ubuntu1404

⁠Services include:

• Mesos Master⁠ - Primary process that manages the collective offering of resources from the Mesos Slaves.
• Consul-Template⁠ - An application that can populate configs from a consul service.
• Logrotate⁠ - A script and application that aid in pruning log files.
• Logstash-Forwarder⁠ - A lightweight log collector and shipper for use with Logstash⁠.
• Redpill⁠ - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
• Rsyslog⁠ - The system logging daemon.

⁠Index

• Usage⁠
• Example Run Command⁠
• Modification and Anatomy of the Project⁠
• Important Environment Variables⁠
• Service Configuration⁠
• Mesos-Master⁠
• Consul-Template⁠
• Logrotate⁠
• Logstash-Forwarder⁠
• Redpill⁠
• Rsyslog⁠
• Troubleshooting⁠

⁠Usage

All mesos commands should be passed via environment variables (please see the example run command⁠ below). For Mesos documentation, please see the configuration docs associated with the release here: mesos@d3717e5⁠.

With the release of Mesos 0.24.0 the Mesos Master (and frameworks) no longer requires running with host networking enabled. This complicates deployments slightly, but overall adds significantly to the portability of the containers and frameworks.

In all forms of deployment, if the Mesos Master container is be accessible by other hosts or processes. There are collection of variables that must be set.

• LIBPROCESS_IP - The IP in which libprocess will bind to (defaults to 0.0.0.0)

• LIBPROCESS_PORT - The port libprocess will use for communication (defaults to 9000)

• LIBPROCESS_ADVERTISE_IP - If set, this will be the 'advertised' or externalized ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication.

• LIBPROCESS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication.

• MESOS_ADVERTISE_IP - If set, this will be the 'advertised' or 'externalized' ip used to reach and communicate with the Mesos Master.

• MESOS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for communication with the Mesos Master.

If deploying WITH host networking, the *_ADVERTISE_* variables may be omitted.

In a local deployment, other than the above the only other variable that must be set is MESOS_ZK.

In a production deployment with high availability, the Master container should be executed with several other variables defined including ENVIRONMENT, MESOS_WORK_DIR, andMESOS_QUORUM.

• ENVIRONMENT - when set to production or development it will enable all services including: mesos-master, logstash-forwarder, and redpill.

• MESOS_ZK - The zookeeper URL used by Mesos for leader election e.g. zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos.

• MESOS_QUORUM - The quorum size of Mesos Registry. It should be equal to the majority of your Mesos Masters. At a bare minimum it should be (# of Mesos Masters/2)+1.

• MESOS_WORK_DIR - The location in which the Mesos registry is stored. A volume should be mounted to the MESOS_WORK_DIR to maintain state across redeployments.

⁠Example Run Command

docker run -d              \
--name master-01           \
--hostname master-01       \
-e ENVIRONMENT=production  \
-e PARENT_HOST=$(hostname) \
-e LIBPROCESS_PORT=9000    \
-e LIBPROCESS_ADVERTISE_PORT=9000      \
-e LIBPROCESS_ADVERTISE_IP=10.10.0.11  \
-e MESOS_ADVERTISE_PORT=5050           \
-e MESOS_ADVERTISE_IP=10.10.0.11       \
-e MESOS_LOG_DIR=/var/log/mesos        \
-e MESOS_LOGGING_LEVEL=WARNING         \
-e MESOS_ZK=zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos \
-e MESOS_QUORUM=2                 \
-e MESOS_CLUSTER=thegrid          \
-e MESOS_WORK_DIR=/var/lib/mesos  \
-p 10.10.0.11:5050:5050           \
-p 10.10.0.11:9000:9000           \
-v /data/mesos:/var/lib/mesos:rw  \
mesos-master

⁠Modification and Anatomy of the Project

File Structure The directory skel in the project root maps to the root of the file system once the container is built. Files and folders placed there will map to their corresponding location within the container.

Init The init script (./init.sh) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.

Supervisord All supervisord configs can be found in /etc/supervisor/conf.d/. Services by default will redirect their stdout to /dev/fd/1 and stderr to /dev/fd/2 allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.

In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.

Logstash-Forwarder The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERVICE_LOGSTASH_FORWARDER_ADDRESS environment variable.

In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.

⁠Important Environment Variables

Below is the minimum list of variables to be aware of when deploying the Mesos Master container.

⁠Defaults

⁠Description

• ENVIRONMENT_INIT - If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables.

• APP_NAME - A brief description of the container. If Logstash-Forwarder is enabled, this will populate the app_name field in the Logstash-Forwarder configuration file.

• ENVIRONMENT - Sets defaults for several other variables based on the current running environment. Please see the environment⁠ section for further information. If logstash-forwarder is enabled, this value will populate the environment field in the logstash-forwarder configuration file.

• PARENT_HOST - The name of the parent host. If Logstash-Forwarder is enabled, this will populate the parent_host field in the Logstash-Forwarder configuration file.

• LIBPROCESS_IP - The IP in which libprocess will bind to (defaults to 0.0.0.0)

• LIBPROCESS_PORT - The port libprocess will use for communication (defaults to 9000)

• LIBPROCESS_ADVERTISE_IP - If set, this will be the 'advertised' or externalized ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication.

• LIBPROCESS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication.

• MESOS_ADVERTISE_IP - If set, this will be the 'advertised' or 'externalized' ip used to reach and communicate with the Mesos Master.

• MESOS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for communication with the Mesos Master.

• MESOS_LOG_DIR - The path to the directory in which Mesos stores its logs.

• MESOS_QUORUM - The quorum size of Mesos Registry. It should be equal to the majority of your Mesos Masters. At a bare minimum it should be (# of Mesos Masters/2)+1.

• MESOS_WORK_DIR - The location in which the Mesos registry is stored. A volume should be mounted to the MESOS_WORK_DIR to maintain state across redeployments.

• MESOS_ZK - The zookeeper URL used by Mesos for leader election e.g. zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos.

• GLOG_max_file_size - The size in Megabytes that the mesos log file(s) will be allowed to grow to before rotation.

• SERVICE_CONSUL_TEMPLATE - * SERVICE_CONSUL_TEMPLATE- Enables or disables the consul-template service. If enabled, it will also enableSERVICE_LOGROTATEandSERVICE_RSYSLOGto handle logging. (**Options:**enabledordisabled`)

• SERVICE_LOGROTATE - Enables or disabled the Logrotate service. This will be set automatically depending on the environment. (Options: enabled or disabled)

• SERVICE_LOGROTATE_INTERVAL - The time in seconds between runs of logrotate or the logrotate script. The default (3600 or 1 hour) is set by default in the logrotate script automatically.

• SERVICE_LOGROTATE_SCRIPT - The path to the script that should be executed instead of logrotate itself to clean up logs.

• SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

• SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

• SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

• SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

• SERVICE_RSYSLOG - Enables of disables the rsyslog service. This is managed by SERVICE_CONSUL_TEMPLATE, but can be enabled/disabled manually. (Options: enabled or disabled)

Environment

• local (default)

• prod|production|dev|development

• debug

* Only set if SERVICE_CONSUL_TEMPLATE is set to enabled.

⁠Service Configuration

⁠Mesos-Master

As stated in the Usage⁠ section, Mesos-slave configuration information can be found in the github docs releated to the Mesos Release: mesos@d3717e5⁠.

The actual mesos start command is passed to supervisor via the SERVICE_MESOS_CMD environment variable, and defaults to mesos-master. It can be overridden by specifying the SERVICE_MESOS_CMD at run time.

⁠Mesos-Master Environment Variables

⁠Defaults

⁠Description

• LIBPROCESS_IP - The IP in which libprocess will bind to (defaults to 0.0.0.0)

• LIBPROCESS_PORT - The port libprocess will use for communication (defaults to 9000)

• LIBPROCESS_ADVERTISE_IP - If set, this will be the 'advertised' or externalized ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication.

• LIBPROCESS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication.

• MESOS_ADVERTISE_IP - If set, this will be the 'advertised' or 'externalized' ip used to reach and communicate with the Mesos Master.

• MESOS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for communication with the Mesos Master.

• MESOS_LOG_DIR - The path to the directory in which Mesos stores its logs.

• MESOS_QUORUM - The quorum size of Mesos Registry. It should be equal to the majority of your Mesos Masters. At a bare minimum it should be (# of Mesos Masters/2)+1.

• MESOS_WORK_DIR - The location in which the Mesos registry is stored. A volume should be mounted to the MESOS_WORK_DIR to maintain state across redeployments.

• MESOS_ZK - The zookeeper URL used by Mesos for leader election e.g. zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos.

• SERVICE_MESOS_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Consul-Template

Provides initial configuration of consul-template. Variables prefixed with CONSUL_TEMPLATE_ will automatically be passed to the consul-template service at runtime, e.g. CONSUL_TEMPLATE_SSL_CA_CERT=/etc/consul/certs/ca.crt becomes -ssl-ca-cert="/etc/consul/certs/ca.crt". If managing the application configuration is handled via file configs, no other variables must be passed at runtime.

⁠Consul-Template Environment Variables

⁠Defaults

⁠Logrotate

The logrotate script is a small simple script that will either call and execute logrotate on a given interval; or execute a supplied script. This is useful for applications that do not perform their own log cleanup.

⁠Logrotate Environment Variables

⁠Defaults

⁠Description

• SERVICE_LOGROTATE - Enables or disables the Logrotate service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

• SERVICE_LOGROTATE_INTERVAL - The time in seconds between run of either the logrotate command or the provided logrotate script. Default is set to 3600 or 1 hour in the script itself.

• SERVICE_LOGROTATE_CONFIG - The path to the logrotate config file. If neither config or script is provided, it will default to /etc/logrotate.conf.

• SERVICE_LOGROTATE_SCRIPT - A script that should be executed on the provided interval. Useful to do cleanup of logs for applications that already handle rotation, or if additional processing is required.

• SERVICE_LOGROTATE_FORCE - If present, passes the 'force' command to logrotate. Will be ignored if a script is provided.

• SERVICE_LOGROTATE_VERBOSE - If present, passes the 'verbose' command to logrotate. Will be ignored if a script is provided.

• SERVICE_LOGROTATE_DEBUG - If present, passed the 'debug' command to logrotate. Will be ignored if a script is provided.

• SERVICE_LOGROTATE_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Logrotate Script Help Text

root@ec58ca7459cb:/opt/scripts# ./logrotate.sh --help
logrotate.sh - Small wrapper script for logrotate.
-i | --interval     The interval in seconds that logrotate should run.
-c | --config       Path to the logrotate config.
-s | --script       A script to be executed in place of logrotate.
-f | --force        Forces log rotation.
-v | --verbose      Display verbose output.
-d | --debug        Enable debugging, and implies verbose output. No state file changes.
-h | --help         This usage text.

⁠Supplied Cleanup Script

The below cleanup script will remove all but the latest 5 rotated logs.

#!/bin/bash

mld=${MESOS_LOG_DIR:-/var/log/mesos}

cd "$mld"

(ls -t | grep 'log.INFO.*'|head -n 5;ls)|sort|uniq -u|grep 'log.INFO.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.ERROR.*'|head -n 5;ls)|sort|uniq -u|grep 'log.ERROR.*'|xargs --no-run-if-empty rm
(ls -t | grep 'log.WARNING.*'|head -n 5;ls)|sort|uniq -u|grep 'log.WARNING.*'|xargs --no-run-if-empty rm

#consul-template uses rsyslog for logging, need to run logrotate to handle that log
if [[ "$SERVICE_CONSUL_TEMPLATE" == "enabled" ]]; then
  /usr/sbin/logrotate /etc/logrotate.conf
fi

⁠Logstash-Forwarder

Logstash-Forwarder is a lightweight application that collects and forwards logs to a logstash server endpoint for further processing. For more information see the Logstash-Forwarder⁠ project.

⁠Logstash-Forwarder Environment Variables

⁠Defaults

⁠Description

• SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

• SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

• SERVICE_LOGSTASH_FORWARDER_ADDRESS - The address of the Logstash server.

• SERVICE_LOGSTASH_FORWARDER_CERT - The path to the Logstash-Forwarder server certificate.

• SERVICE_LOGSTASH_FORWARDER_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Redpill

Redpill is a small script that performs status checks on services managed through supervisor. In the event of a failed service (FATAL) Redpill optionally runs a cleanup script and then terminates the parent supervisor process.

⁠Redpill Environment Variables

⁠Defaults

⁠Description

• SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section. (Options: enabled or disabled)

• SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

• SERVICE_REDPILL_INTERVAL - The interval in which Redpill polls supervisor for status checks. (Default for the