Prometheus⁠ on Mesosphere DC/OS⁠

Intro

This runs Prometheus on DC/OS (1.8+). server.json contains the service definition for Prometheus itself. node_exporter.json contains the service definition for node_exporter. I'm running node_exporter inside a Mesos (cgroups) container so that it sees all of the hosts filesystems without any need for priviliges or translation.

To make life easier I also created a group.json that includes the Prometheus Server, Node Exporter, cAdvisor⁠, Grafana Dashboard⁠ and an authentication proxy⁠ which will add Basic Auth to the Server's WebUI. The group assumes you're running Marathon-LB⁠ on your DC/OS and exports Marathon-LB labels.

To get started just install the group as shown below.

Usage

Install using

$ dcos marathon group add https://raw.githubusercontent.com/lloesche/prometheus-dcos/master/group.json
$ dcos marathon app update /prometheus/node-exporter instances=7000 # however many agents you have in your cluster

Important: Once the apps are deployed make sure to update all Environment Variables with something useful. Alternatively download group.json and modify them directly before deploying to DC/OS.

When working with the group.json you'll want to adjust the following variables and labels:

Connections

Why file_sd based discovery?

Prometheus supports DNS based service discovery. Given a Mesos-DNS SRV record like _node-exporter.prometheus._tcp.marathon.mesos it will find the list of node_exporter nodes and poll them. However it'll result in instance names like

node-exporter.prometheus-6ms1y-s1.marathon.mesos:14181
node-exporter.prometheus-54eio-s0.marathon.mesos:12227
node-exporter.prometheus-1e1ow-s2.marathon.mesos:31798

which is not very useful. Also the Mesos scheduler will assign a random port resource.

So after a discussion on the mailing list⁠ it turned out that Prometheus can't relabel the instance with the node's IP address since name resolution happens after relabeling. It was suggested to use the file_sd based discovery method instead. This is what the srv2file_sd helper is for. It performs the same SRV and A record lookup and instead of the hostname writes the node's IP addres into the targets file. There's also relabeling taking place to replace the random port number with the node_exporter standard port 9100 so that when a node_exporter is restarted on a different port it's data is still associated with the same node.

Environment Variables

To produce the $RULES env variable it can be handy to use something like

$ cat prometheus.rules | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'

Building the SRV lookup helper

To run the srv2file_sd helper tool inside the minimal prom/prometheus Docker container I statically linked it. To do so yourself install musl libc⁠ and compile using:

$ CC=/usr/local/musl/bin/musl-gcc go build --ldflags '-linkmode external -extldflags "-static"' srv2file_sd.go

Bugs

All this was hacked up in an afternoon. Surely there's bugs. If you find any submit a PR or open an issue.

TODO

• perform A lookups in parallel instead of looping over all hosts sequentially