pixelfederation/mesos-dns repository overview

⁠- Mesos DNS -

An Ubuntu based container - built for running the Mesos-DNS support service. It comes bundled with Logstash-Forwarder, and is managed by Supervisord. All parameters are controlled through environment variables, with some settings auto-configured based on the environment.

⁠Version Information:

• Container Release: 1.3.2
• Mesos-DNS: 0.6.0

⁠Services Include:

• Mesos-DNS⁠ - A small application that provides DNS as a method of service discovery for applications launched via Mesos and it's associated frameworks.
• Consul-Template⁠ - An application that can populate configs from a consul service.
• Logrotate⁠ - A script and application that aid in pruning log files.
• Logstash-Forwarder⁠ - A lightweight log collector and shipper for use with Logstash⁠.
• Redpill⁠ - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
• Rsyslog⁠ - The system logging daemon.

⁠Index

• Usage⁠
• Example Run Command⁠
• Example Marathon App Definition⁠
• Modification and Anatomy of the Project⁠
• Important Environment Variables⁠
• Service Configuration⁠
• Mesos-DNS⁠
• Consul-Template⁠
• Logrotate⁠
• Logstash-Forwarder⁠
• Redpill⁠
• Rsyslog⁠
• Troubleshooting⁠

⁠Usage

The Mesos-DNS container is fairly easy to get going. Outside of running the container with host networking and specifying the ENVIRONMENT, the only required variables that must be defined are MESOSDNS_ZK or MESOSDNS_MASTERS_###, and MESOSDNS_RESOLVERS_###.

• MESOSDNS_ZK - The Zookeeper Mesos uri. e.g. zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos

• MESOSDNS_MASTERS_### - The address of the Mesos masters in the form of ip:port. e.g. MESOSDNS_MASTERS_1=10.10.0.11:5050. If MESOSDNS_ZK is already set, these are not necessary.

• MESOSDNS_RESOLVERS_### - The IP of an upstream DNS server. More than one can be specified using the same syntax as MESOSDNS_MASTERS_### e.g. MESOSDNS_RESOLVERS_1=8.8.8.8

With MESOSDNS_AUTOCONF enabled. The init script will assemble a Mesos-DNS config file based on environment variables beginning with the prefix MESOSDNS_. The environment variable names correspond with their associated Mesos-DNS config option. e.g. the option "domain": "mesos" would map to MESOSDNS_DOMAIN="mesos".

For options that would be represented by an array, they can be passed by adding an _ followed by a number from 0-999. e.g. "masters": ["10.10.0.11:5050", "10.10.0.12:5050", "10.10.0.13:5050"] would map to:

MESOSDNS_MASTERS_1="10.10.0.11:5050"
MESOSDNS_MASTERS_2="10.10.0.12:5050"
MESOSDNS_MASTERS_3="10.10.0.13:5050"

Alternatively, if you already have a Mesos-DNS config, MESOSNS_AUTOCONF can be disabled and all that is required is supplying the path in the MESOSDNS_CONF variable.

For a full list of Mesos-DNS options, please see the Mesos-DNS Service⁠ section, or visit Mesos-DNS⁠ main documentation page.

Marathon Deployments For Marathon based deployments; if healthchecks are going to be used - the Mesos-DNS webserver should be enabled and the Marathon healthcheck itself must be a COMMAND based healthcheck. This is the case for anything with host based networking.

⁠Example Run Command

docker run -d --net=host \
-e ENVIRONMENT=production \
-e PARENT_HOST=$(hostname) \
-e MESOSDNS_ZK="zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos" \
-e MESOSDNS_MASTERS_1="10.10.0.11:5050" \
-e MESOSDNS_MASTERS_2="10.10.0.12:5050" \
-e MESOSDNS_MASTERS_3="10.10.0.13:5050" \
-e MESOSDNS_RESOLVERS_1="8.8.8.8" \
-e MESOSDNS_RESOLVERS_2="8.8.4.4" \
-e MESOSDNS_DOMAIN="mesos" \
-e MESOSDNS_REFRESHSECONDS=60 \
-e MESOSDNS_TTL=60 \
-e MESOSDNS_TIMEOUT=5 \
-e MESOSDNS_PORT=53 \
-e MESOSDNS_HTTPPORT=8123 \
mesos-dns

⁠Example Marathon App Definition

{
    "id": "/mesos-dns",
    "instances": 1,
    "cpus": 0.5,
    "mem": 512,
    "constraints": [
        [
            "hostname",
            "CLUSTER",
            "10.10.111"
        ]
    ],
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "registry.address/mesos-dns",
            "network": "HOST"
        }
    },
    "env": {
        "ENVIRONMENT": "production",
        "MESOSDNS_ZK": "zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos",
        "MESOSDNS_MASTERS_1": "10.10.0.11:5050",
        "MESOSDNS_MASTERS_2": "10.10.0.12:5050",
        "MESOSDNS_MASTERS_3": "10.10.0.13:5050",
        "MESOSDNS_RESOLVERS_1": "8.8.8.8",
        "MESOSDNS_RESOLVERS_2": "8.8.4.4",
        "MESOSDNS_DOMAIN": "mesos",
        "MESOSDNS_REFRESHSECONDS": "60",
        "MESOSDNS_TTL": "60",
        "MESOSDNS_TIMEOUT": "5",
        "MESOSDNS_PORT": "53",
        "MESOSDNS_HTTPORT": "8123"
    },
    "healthChecks": [
        {
            "protocol": "COMMAND",
            "command": {
                "value": "curl -f -X GET http://$HOST:8123/v1/version"
            },
            "gracePeriodSeconds": 30,
            "timeoutSeconds": 60,
            "maxConsecutiveFailures": 5
        }
    ],
    "backoffSeconds": 1,
    "backoffFactor": 1.5,
    "maxLaunchDelaySeconds": 3600,
    "uris": [
        "file: ///docker.tar.gz"
    ]
}

⁠Modification and Anatomy of the Project

File Structure The directory skel in the project root maps to the root of the filesystem once the container is built. Files and folders placed there will map to their corresponding location within the container.

Init The init script (./init.sh) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.

Supervisord All supervisord configs can be found in /etc/supervisor/conf.d/. Services by default will redirect their stdout to /dev/fd/1 and stderr to /dev/fd/2 allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.

In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.

Logstash-Forwarder The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERICE_LOGSTASH_FORWARDER_ADDRESS environment variable.

In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.

⁠Important Environment Variables

Below is the minimum list of variables to be aware of when deploying the Mesos-DNS container.

⁠Defaults

* SERVICE_RSYSLOG is automatically enabled if SERVICE_CONSUL_TEMPLATE is enabled to ensure logging.

⁠Description

• ENVIRONMENT_INIT - If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables.

• APP_NAME - A brief description of the container. If Logstash-Forwarder is enabled, this will populate the app_name field in the Logstash-Forwarder configuration file.

• ENVIRONMENT - Sets defaults for several other variables based on the current running environment. Please see the environment⁠ section for further information. If logstash-forwarder is enabled, this value will populate the environment field in the logstash-forwarder configuration file.

• PARENT_HOST - The name of the parent host. If Logstash-Forwarder is enabled, this will populate the parent_host field in the Logstash-Forwarder configuration file.

• MESOSDNS_AUTOCONF - If this is enabled, the init script will attempt to autogenerate a config based on additional environment variables being passed. Please see the Usage⁠ section for more information.

• MESOSDNS_CONF - The path to the Mesos-DNS configuration file (json format)

• MESOSDNS_MASTERS_### - All MESOSDNS_MASTERS_### are converted to comma separated list with the IP address and port number for the master(s) in the Mesos cluster. Mesos-DNS will automatically find the leading master at any point in order to retrieve state about running tasks. If there is no leading master or the leading master is not responsive, Mesos-DNS will continue serving DNS requests based on stale information about running tasks. The masters field is required.

• MESOSDNS_ZK -MESOSDNS_ZK is a link to the Zookeeper instances on the Mesos cluster. Its format is zk://host1:port1,host2:port2/mesos/, where the number of hosts can be one or more. The default port for Zookeeper is 2181. Mesos-DNS will monitor the Zookeeper instances to detect the current leading master.

• MESOSDNS_RESOLVERS_### - ALL MESOSDNS_RESOLVERS_### are converted to a comma separated list with the IP addresses of external DNS servers that Mesos-DNS will contact to resolve any DNS requests outside the domain. We recommend that you list the nameservers specified in the /etc/resolv.conf on the server Mesos-DNS is running. Alternatively, you can list 8.8.8.8, which is the Google public DNS address. The resolvers field is required.

• MESOSDNS_LISTENER - It is the IP address of Mesos-DNS. In SOA replies, Mesos-DNS identifies hostname mesos-dns.domain as the primary nameserver for the domain. It uses this IP address in an A record for mesos-dns.domain. The default value is "0.0.0.0", which instructs Mesos-DNS to create an A record for every IP address associated with a network interface on the server that runs the Mesos-DNS process.

• SERVICE_CONSUL_TEMPLATE - Enables or disables the consul-template service. (Options: enabled or disabled)

• SERVICE_LOGROTATE - Enables or disabled the Logrotate service. This will be set automatically depending on the environment. (Options: enabled or disabled)

• SERVICE_LOGROTATE_INTERVAL - The time in seconds between runs of logrotate or the logrotate script. The default (3600 or 1 hour) is set by default in the logrotate script automatically.

• SERVICE_LOGROTATE_SCRIPT - The path to the script that should be executed instead of logrotate itself to clean up logs.

• SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

• SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

• SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options: enabled or disabled)

• SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

• SERVICE_RSYSLOG - Enables of disables the rsyslog service. This is managed by SERVICE_CONSUL_TEMPLATE, but can be enabled/disabled manually.

⁠Environment

• local (default)

• prod|production|dev|development

• debug

* Only set if SERVICE_CONSUL_TEMPLATE is set to enabled.

⁠Service Configurations

⁠Mesos-DNS

⁠Description

• MESOSDNS_AUTOCONF - If this is enabled, the init script will attempt to autogenerate a config based on additional environment variables being passed. Please see the Usage⁠ section for more information.

• MESOSDNS_CONF - The path to the Mesos-DNS configuration file (json format)

• MESOSDNS_OPTS - Additional Options to pass to Mesos-DNS. This generally control logging options. For more information see the Mesos-DNS Help text below.

• SERVICE_MESOSDNS_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation⁠ for further information.

⁠Mesos-DNS configuration options:

Notes:

1. These descriptions are taken right from the Mesos-DNS documentation page⁠ with a few modifications to fit how they're used in this project. They are listed here for convenience.
2. It is sufficient to specify just one of the MESOSDNS_ZK or MESOSDNS_MASTERS_###. If both are defined, Mesos-DNS will first attempt to detect the leading master through Zookeeper. If Zookeeper is not responding, it will fall back to using the masters field. Both zk and master fields are static. To update them you need to restart Mesos-DNS. We recommend you use the zk field since this allows the dynamic addition to Mesos masters.

⁠Description

• MESOSDNS_ZK -MESOSDNS_ZK is a link to the Zookeeper instances on the Mesos cluster. Its format is zk://host1:port1,host2:port2/mesos/, where the number of hosts can be one or more. The default port for Zookeeper is 2181. Mesos-DNS will monitor the Zookeeper instances to detect the current leading master.

• MESOSDNS_ZKDETECTIONTIMEOUT - Time in seconds for Zookeeper to report a new Mesos leading master. If this threshold is crossed, Mesos-DNS will exit. Default value is 30.

• MESOSDNS_MASTERS_### - All MESOSDNS_MASTERS_### are converted to comma separated list with the IP address and port number for the master(s) in the Mesos cluster. Mesos-DNS will automatically find the leading master at any point in order to retrieve state about running tasks. If there is no leading master or the leading master is not responsive, Mesos-DNS will continue serving DNS requests based on stale information about running tasks. The masters field is required.

• MESOSDNS_REFRESHSECONDS - The frequency at which Mesos-DNS updates DNS records based on information retrieved from the Mesos master. The default value is 60 seconds.

• MESOSDNS_TTL - The time to live value for DNS records served by Mesos-DNS, in seconds. It allows caching of the DNS record for a period of time in order to reduce DNS request rate. MESOSDNS_TTL should be equal or larger than MESOSDNS_REFRESHSECONDS. The default value is 60 seconds.

• MESOSDNS_DOMAIN - Is the domain name for the Mesos cluster. The domain name can use characters [a-z, A-Z, 0-9], - if it is not the first or last character of a domain portion, and . as a separator of the textual portions of the domain name. We recommend you avoid valid top-level domain names. The default value is mesos.

• MESOSDNS_PORT - Is the port number that Mesos-DNS monitors for incoming DNS requests. Requests can be sent over TCP or UDP. We recommend you use port 53 as several applications assume that the DNS server listens to this port. The default value is 53.

• MESOSDNS_RESOLVERS_### - ALL MESOSDNS_RESOLVERS_### are converted to a json list with the IP addresses of external DNS servers that Mesos-DNS will contact to resolve any DNS requests outside the domain. We recommend that you list the nameservers specified in the /etc/resolv.conf on the server Mesos-DNS is running. Alternatively, you can list 8.8.8.8, which is the Google public DNS address. The resolvers field is required.

• MESOSDNS_TIMEOUT - Is the timeout threshold, in seconds, for connections and requests to external DNS requests. The default value is 5 seconds.

• MESOSDNS_HTTPON - Is a boolean field that controls whether Mesos-DNS listens for HTTP requests or not. The default value is true.

• MESOSDNS_DNSON - Is a boolean field that controls whether Mesos-DNS listens for DNS requests or not. The default value is true.

• MESOSDNS_HTTPPORT - Is the port number that Mesos-DNS monitors for incoming HTTP requests. The default value is 8123.

• MESOSDNS_EXTERNALON - Is a boolean field that controls whether Mesos-DNS serves requests outside of the Mesos domain. The default value is true.

• MESOSDNS_LISTENER - It is the IP address of Mesos-DNS. In SOA replies, Mesos-DNS identifies hostname mesos-dns.domain as the primary nameserver for the domain. It uses this IP address in an A record for mesos-dns.domain. The default value is "0.0.0.0", which instructs Mesos-DNS to create an A record for every IP address associated with a network interface on the server that runs the Mesos-DNS process.

• MESOSDNS_SOAMNAME - Is the MNAME field in the SOA record for the Mesos domain. It is the primary or master name server. The default value is ns1.mesos.

• MESOSDNS_SOARNAME - Is the RNAME field in the SOA record for the Mesos domain. The format is mailbox.domain, using a . instead of @. For example, if the email address is [email protected], the email field should be root.mesos-dns.mesos. For details, see the RFC-1035⁠. The default value is root.ns1.mesos.

• MESOSDNS_SOAREFRESH - Is the REFRESH field in the SOA record for the Mesos domain. For details, see the RFC-1035⁠. The default value is 60.

• MESOSDNS_RETRY - Is the RETRY field in the SOA record for the Mesos domain. For details, see the RFC-1035⁠. The default value is 600.

• MESOSDNS_EXPIRE - Is the EXPIRE field in the SOA record for the Mesos domain. For details, see the RFC-1035⁠. The default value is 86400.

• MESOSDNS_SOAMINTTL - Is the minimum TTL field in the SOA record for the Mesos domain. For details, see the RFC-1035⁠. The default value is 60.

• MESOSDNS_STATETIMEOUTSECONDS - The time, in seconds that Mesos-DNS will wait for the Mesos master to respond to it's requests for state.json. The default value is 300.

• MESOSDNS_RECURSEON - Controls if the DNS replies for names in the Mesos domain will indicate that recursion is available. The default value is true.

• MESOSDNS_ENFORCERCF952 - Enables an older stricter set of rules for DNS labels. For more information, see RFC-952⁠. Default value is `f