[{"_1":2,"_160":-5,"_161":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_139":140},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_87":-5,"_88":89,"_128":-5,"_129":11,"_130":21,"_131":11,"_132":-5,"_133":11,"_134":-5,"_135":136,"_137":90,"_138":-7},"canonicalUrl","https://hub.docker.com/r/dhileepbalaji/node-exporter/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","dhileepbalaji","reponame","node-exporter","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":35,"_48":11,"_49":50,"_51":52,"_53":54,"_56":35,"_32":33,"_57":58,"_59":44,"_60":61,"_62":63,"_64":65,"_66":-5,"_67":61,"_68":69,"_70":11,"_71":33,"_72":73,"_78":11,"_79":80,"_84":33,"_85":86},"categories",[],"content_types",[44],"image","date_registered","2020-05-13T10:42:58.377388Z","description","is_private","last_updated","2020-07-28T07:54:32.213365Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[55],"application/vnd.docker.container.image.v1+json","name","pull_count",905,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# swarmprom\n\nSwarmprom is a starter kit for Docker Swarm monitoring with [Prometheus](https://prometheus.io/),\n[Grafana](http://grafana.org/),\n[cAdvisor](https://github.com/google/cadvisor),\n[Node Exporter](https://github.com/prometheus/node_exporter),\n[Alert Manager](https://github.com/prometheus/alertmanager)\nand [Unsee](https://github.com/cloudflare/unsee).\n\n## Install\n\nClone this repository and run the monitoring stack:\n\n```bash\n$ git clone https://github.com/dhileepbalaji/swarmprom.git\n$ cd swarmprom\n\nADMIN_USER=admin \\\nADMIN_PASSWORD=admin \\\nSLACK_URL=https://hooks.slack.com/services/TOKEN \\\nSLACK_CHANNEL=devops-alerts \\\nSLACK_USER=alertmanager \\\ndocker stack deploy -c docker-compose.yml mon\n```\n\nPrerequisites:\n\n* Minimum Docker CE 17.09.0-ce or Docker EE 17.06.2-ee-3\n* Swarm cluster with one manager and a worker node\n* Docker engine experimental enabled and metrics address set to `0.0.0.0:9323`\n\nServices:\n\n* prometheus (metrics database) `http://:9090`\n* grafana (visualize metrics) `http://:3000`\n* node-exporter (host metrics collector)\n* cadvisor (containers metrics collector)\n* dockerd-exporter (Docker daemon metrics collector, requires Docker experimental metrics-addr to be enabled)\n* alertmanager (alerts dispatcher) `http://:9093`\n* unsee (alert manager dashboard) `http://:9094`\n\n\n\n## Install with Traefik and HTTPS\n\n### Requisites\n\nThese instructions assume you already have Traefik set up following that guide above, in short:\n\n* With automatic HTTPS certificate generation.\n* A Docker Swarm network `traefik`.\n* Filtering to only serve containers with a tag `traefik-prod`.\n\n### Instructions\n\n* Clone this repository and enter into the directory:\n\n```bash\n$ git clone https://github.com/dhileepbalaji/swarmprom.git\n$ cd swarmprom\n```\n\n* Set and export an `ADMIN_USER` environment variable:\n\n```bash\nexport ADMIN_USER=admin\n```\n\n* Set and export an `ADMIN_PASSWORD` environment variable:\n\n\n```bash\nexport ADMIN_PASSWORD=changethis\n```\n\n* Set and export a hashed version of the `ADMIN_PASSWORD` using `openssl`, it will be used by Traefik's HTTP Basic Auth for most of the services:\n\n```bash\nexport HASHED_PASSWORD=$(openssl passwd -apr1 $ADMIN_PASSWORD)\n```\n\n* You can check the contents with:\n\n```bash\necho $HASHED_PASSWORD\n```\n\nit will look like:\n\n```\n$apr1$89eqM5Ro$CxaFELthUKV21DpI3UTQO.\n```\n\n* Create and export an environment variable `DOMAIN`, e.g.:\n\n```bash\nexport DOMAIN=example.com\n```\n\nand make sure that the following sub-domains point to your Docker Swarm cluster IPs:\n\n* `grafana.example.com`\n* `alertmanager.example.com`\n* `unsee.example.com`\n* `prometheus.example.com`\n\n(and replace `example.com` with your actual domain).\n\n**Note**: You can also use a subdomain, like `swarmprom.example.com`. Just make sure that the subdomains point to (at least one of) your cluster IPs. Or set up a wildcard subdomain (`*`).\n\n* Set and export an environment variable with the tag used by Traefik public to filter services (by default, it's `traefik-prod`):\n\n```bash\nexport TRAEFIK_PUBLIC_TAG=traefik-prod\n```\n\n* If you are using Slack and want to integrate it, set the following environment variables:\n\n```bash\nexport SLACK_URL=https://hooks.slack.com/services/TOKEN\nexport SLACK_CHANNEL=devops-alerts\nexport SLACK_USER=alertmanager\n```\n\n**Note**: by using `export` when declaring all the environment variables above, the next command will be able to use them.\n\n* Deploy the Traefik version of the stack:\n\n\n```bash\ndocker stack deploy -c docker-compose.traefik.yml monitoring\n```\n\nTo test it, go to each URL:\n\n* `https://grafana.example.com`\n* `https://alertmanager.example.com`\n* `https://unsee.example.com`\n* `https://prometheus.example.com`\n\n\n## Setup Grafana\n\nNavigate to `http://:3000` and login with user ***admin*** password ***admin***.\nYou can change the credentials in the compose file or\nby supplying the `ADMIN_USER` and `ADMIN_PASSWORD` environment variables at stack deploy.\n\nSwarmprom Grafana is preconfigured with two dashboards and Prometheus as the default data source:\n\n* Name: Prometheus\n* Type: Prometheus\n* Url: http://prometheus:9090\n* Access: proxy\n\nAfter you login, click on the home drop down, in the left upper corner and you'll see the dashboards there.\n\n***Docker Swarm Nodes Dashboard***\n\n![Nodes](https://raw.githubusercontent.com/stefanprodan/swarmprom/master/grafana/screens/swarmprom-nodes-dash-v3.png)\n\nURL: `http://:3000/dashboard/db/docker-swarm-nodes`\n\nThis dashboard shows key metrics for monitoring the resource usage of your Swarm nodes and can be filtered by node ID:\n\n* Cluster up-time, number of nodes, number of CPUs, CPU idle gauge\n* System load average graph, CPU usage graph by node\n* Total memory, available memory gouge, total disk space and available storage gouge\n* Memory usage graph by node (used and cached)\n* I/O usage graph (read and write Bps)\n* IOPS usage (read and write operation per second) and CPU IOWait\n* Running containers graph by Swarm service and node\n* Network usage graph (inbound Bps, outbound Bps)\n* Nodes list (instance, node ID, node name)\n\n***Docker Swarm Services Dashboard***\n\n![Nodes](https://raw.githubusercontent.com/stefanprodan/swarmprom/master/grafana/screens/swarmprom-services-dash-v3.png)\n\nURL: `http://:3000/dashboard/db/docker-swarm-services`\n\nThis dashboard shows key metrics for monitoring the resource usage of your Swarm stacks and services, can be filtered by node ID:\n\n* Number of nodes, stacks, services and running container\n* Swarm tasks graph by service name\n* Health check graph (total health checks and failed checks)\n* CPU usage graph by service and by container (top 10)\n* Memory usage graph by service and by container (top 10)\n* Network usage graph by service (received and transmitted)\n* Cluster network traffic and IOPS graphs\n* Docker engine container and network actions by node\n* Docker engine list (version, node id, OS, kernel, graph driver)\n\n***Prometheus Stats Dashboard***\n\n![Nodes](https://raw.githubusercontent.com/stefanprodan/swarmprom/master/grafana/screens/swarmprom-prometheus-dash-v3.png)\n\nURL: `http://:3000/dashboard/db/prometheus`\n\n* Uptime, local storage memory chunks and series\n* CPU usage graph\n* Memory usage graph\n* Chunks to persist and persistence urgency graphs\n* Chunks ops and checkpoint duration graphs\n* Target scrapes, rule evaluation duration, samples ingested rate and scrape duration graphs\n\n\n## Prometheus service discovery\n\nIn order to collect metrics from Swarm nodes you need to deploy the exporters on each server.\nUsing global services you don't have to manually deploy the exporters. When you scale up your\ncluster, Swarm will launch a cAdvisor, node-exporter and dockerd-exporter instance on the newly created nodes.\nAll you need is an automated way for Prometheus to reach these instances.\n\nRunning Prometheus on the same overlay network as the exporter services allows you to use the DNS service\ndiscovery. Using the exporters service name, you can configure DNS discovery:\n\n```yaml\nscrape_configs:\n - job_name: 'node-exporter'\n dns_sd_configs:\n - names:\n - 'tasks.node-exporter'\n type: 'A'\n port: 9100\n - job_name: 'cadvisor'\n dns_sd_configs:\n - names:\n - 'tasks.cadvisor'\n type: 'A'\n port: 8080\n - job_name: 'dockerd-exporter'\n dns_sd_configs:\n - names:\n - 'tasks.dockerd-exporter'\n type: 'A'\n port: 9323\n```\n\nWhen Prometheus runs the DNS lookup, Docker Swarm will return a list of IPs for each task.\nUsing these IPs, Prometheus will bypass the Swarm load-balancer and will be able to scrape each exporter\ninstance.\n\nThe problem with this approach is that you will not be able to tell which exporter runs on which node.\nYour Swarm nodes' real IPs are different from the exporters IPs since exporters IPs are dynamically\nassigned by Docker and are part of the overlay network.\nSwarm doesn't provide any records for the tasks DNS, besides the overlay IP.\nIf Swarm provides SRV records with the nodes hostname or IP, you can re-label the source\nand overwrite the overlay IP with the real IP.\n\nIn order to tell which host a node-exporter instance is running, I had to create a prom file inside\nthe node-exporter containing the hostname and the Docker Swarm node ID.\n\nWhen a node-exporter container starts `node-meta.prom` is generated with the following content:\n\n```bash\n\"node_meta{node_id=\\\"$NODE_ID\\\", node_name=\\\"$NODE_NAME\\\"} 1\"\n```\n\nThe node ID value is supplied via `{{.Node.ID}}` and the node name is extracted from the `/etc/hostname`\nfile that is mounted inside the node-exporter container.\n\n```yaml\n node-exporter:\n image: stefanprodan/swarmprom-node-exporter\n environment:\n - NODE_ID={{.Node.ID}}\n volumes:\n - /etc/hostname:/etc/nodename\n command:\n - '-collector.textfile.directory=/etc/node-exporter/'\n```\n\nUsing the textfile command, you can instruct node-exporter to collect the `node_meta` metric.\nNow that you have a metric containing the Docker Swarm node ID and name, you can use it in promql queries.\n\nLet's say you want to find the available memory on each node, normally you would write something like this:\n\n```\nsum(node_memory_MemAvailable) by (instance)\n\n{instance=\"10.0.0.5:9100\"} 889450496\n{instance=\"10.0.0.13:9100\"} 1404162048\n{instance=\"10.0.0.15:9100\"} 1406574592\n```\n\nThe above result is not very helpful since you can't tell what Swarm node is behind the instance IP.\nSo let's write that query taking into account the node_meta metric:\n\n```sql\nsum(node_memory_MemAvailable * on(instance) group_left(node_id, node_name) node_meta) by (node_id, node_name)\n\n{node_id=\"wrdvtftteo0uaekmdq4dxrn14\",node_name=\"swarm-manager-1\"} 889450496\n{node_id=\"moggm3uaq8tax9ptr1if89pi7\",node_name=\"swarm-worker-1\"} 1404162048\n{node_id=\"vkdfx99mm5u4xl2drqhnwtnsv\",node_name=\"swarm-worker-2\"} 1406574592\n```\n\nThis is much better. Instead of overlay IPs, now I can see the actual Docker Swarm nodes ID and hostname. Knowing the hostname of your nodes is useful for alerting as well.\n\nYou can define an alert when available memory reaches 10%. You also will receive the hostname in the alert message\nand not some overlay IP that you can't correlate to a infrastructure item.\n\nMaybe you are wondering why you need the node ID if you have the hostname. The node ID will help you match\nnode-exporter instances to cAdvisor instances. All metrics exported by cAdvisor have a label named `container_label_com_docker_swarm_node_id`,\nand this label can be used to filter containers metrics by Swarm nodes.\n\nLet's write a query to find out how many containers are running on a Swarm node.\nKnowing from the `node_meta` metric all the nodes IDs you can define a filter with them in Grafana.\nAssuming the filter is `$node_id` the container count query should look like this:\n\n```\ncount(rate(container_last_seen{container_label_com_docker_swarm_node_id=~\"$node_id\"}[5m]))\n```\n\nAnother use case for node ID is filtering the metrics provided by the Docker engine daemon.\nDocker engine doesn't have a label with the node ID attached on every metric, but there is a `swarm_node_info`\nmetric that has this label. If you want to find out the number of failed health checks on a Swarm node\nyou would write a query like this:\n\n```\nsum(engine_daemon_health_checks_failed_total) * on(instance) group_left(node_id) swarm_node_info{node_id=~\"$node_id\"})\n```\n\nFor now the engine metrics are still experimental. If you want to use dockerd-exporter you have to enable\nthe experimental feature and set the metrics address to `0.0.0.0:9323`.\n\nIf you are running Docker with systemd create or edit\n/etc/systemd/system/docker.service.d/docker.conf file like so:\n\n```\n[Service]\nExecStart=\nExecStart=/usr/bin/dockerd \\\n --storage-driver=overlay2 \\\n --dns 8.8.4.4 --dns 8.8.8.8 \\\n --experimental=true \\\n --metrics-addr 0.0.0.0:9323\n```\n\nApply the config changes with `systemctl daemon-reload && systemctl restart docker` and\ncheck if the docker_gwbridge ip address is 172.18.0.1:\n\n```bash\nip -o addr show docker_gwbridge\n```\n\nReplace 172.18.0.1 with your docker_gwbridge address in the compose file:\n\n```yaml\n dockerd-exporter:\n image: stefanprodan/caddy\n environment:\n - DOCKER_GWBRIDGE_IP=172.18.0.1\n```\n\nCollecting Docker Swarm metrics with Prometheus is not a smooth process, and\nbecause of `group_left` queries tend to become more complex.\nIn the future I hope Swarm DNS will contain the SRV record for hostname and Docker engine\nmetrics will expose container metrics replacing cAdvisor all together.\n\n## Configure Prometheus\n\nI've set the Prometheus retention period to 24h, you can change these values in the\ncompose file or using the env variable `PROMETHEUS_RETENTION`.\n\n```yaml\n prometheus:\n image: stefanprodan/swarmprom-prometheus\n command:\n - '-storage.tsdb.retention=24h'\n deploy:\n resources:\n limits:\n memory: 2048M\n reservations:\n memory: 1024M\n```\n\nWhen using host volumes you should ensure that Prometheus doesn't get scheduled on different nodes. You can\npin the Prometheus service on a specific host with placement constraints.\n\n```yaml\n prometheus:\n image: stefanprodan/swarmprom-prometheus\n volumes:\n - prometheus:/prometheus\n deploy:\n mode: replicated\n replicas: 1\n placement:\n constraints:\n - node.labels.monitoring.role == prometheus\n```\n\n## Configure alerting\n\nThe Prometheus swarmprom comes with the following alert rules:\n\n***Swarm Node CPU Usage***\n\nAlerts when a node CPU usage goes over 80% for five minutes.\n\n```\nALERT node_cpu_usage\n IF 100 - (avg(irate(node_cpu{mode=\"idle\"}[1m]) * on(instance) group_left(node_name) node_meta * 100) by (node_name)) > 80\n FOR 5m\n LABELS { severity=\"warning\" }\n ANNOTATIONS {\n summary = \"CPU alert for Swarm node '{{ $labels.node_name }}'\",\n description = \"Swarm node {{ $labels.node_name }} CPU usage is at {{ humanize $value}}%.\",\n }\n```\n***Swarm Node Memory Alert***\n\nAlerts when a node memory usage goes over 80% for five minutes.\n\n```\nALERT node_memory_usage\n IF sum(((node_memory_MemTotal - node_memory_MemAvailable) / node_memory_MemTotal) * on(instance) group_left(node_name) node_meta * 100) by (node_name) > 80\n FOR 5m\n LABELS { severity=\"warning\" }\n ANNOTATIONS {\n summary = \"Memory alert for Swarm node '{{ $labels.node_name }}'\",\n description = \"Swarm node {{ $labels.node_name }} memory usage is at {{ humanize $value}}%.\",\n }\n```\n***Swarm Node Disk Alert***\n\nAlerts when a node storage usage goes over 85% for five minutes.\n\n```\nALERT node_disk_usage\n IF ((node_filesystem_size{mountpoint=\"/rootfs\"} - node_filesystem_free{mountpoint=\"/rootfs\"}) * 100 / node_filesystem_size{mountpoint=\"/rootfs\"}) * on(instance) group_left(node_name) node_meta > 85\n FOR 5m\n LABELS { severity=\"warning\" }\n ANNOTATIONS {\n summary = \"Disk alert for Swarm node '{{ $labels.node_name }}'\",\n description = \"Swarm node {{ $labels.node_name }} disk usage is at {{ humanize $value}}%.\",\n }\n```\n\n***Swarm Node Disk Fill Rate Alert***\n\nAlerts when a node storage is going to remain out of free space in six hours.\n\n```\nALERT node_disk_fill_rate_6h\n IF predict_linear(node_filesystem_free{mountpoint=\"/rootfs\"}[1h], 6*3600) * on(instance) group_left(node_name) node_meta < 0\n FOR 1h\n LABELS { severity=\"critical\" }\n ANNOTATIONS {\n summary = \"Disk fill alert for Swarm node '{{ $labels.node_name }}'\",\n description = \"Swarm node {{ $labels.node_name }} disk is going to fill up in 6h.\",\n }\n```\n\nYou can add alerts to\n[swarm_node](https://github.com/stefanprodan/swarmprom/blob/master/prometheus/rules/swarm_node.rules)\nand [swarm_task](https://github.com/stefanprodan/swarmprom/blob/master/prometheus/rules/swarm_task.rules)\nfiles and rerun stack deploy to update them. Because these files are mounted inside the Prometheus\ncontainer at run time as [Docker configs](https://docs.docker.com/engine/swarm/configs/)\nyou don't have to bundle them with the image.\n\nThe Alertmanager swarmprom image is configured with the Slack receiver.\nIn order to receive alerts on Slack you have to provide the Slack API url,\nusername and channel via environment variables:\n\n```yaml\n alertmanager:\n image: stefanprodan/swarmprom-alertmanager\n environment:\n - SLACK_URL=${SLACK_URL}\n - SLACK_CHANNEL=${SLACK_CHANNEL}\n - SLACK_USER=${SLACK_USER}\n```\n\nYou can install the `stress` package with apt and test out the CPU alert, you should receive something like this:\n\n![Alerts](https://raw.githubusercontent.com/stefanprodan/swarmprom/master/grafana/screens/alertmanager-slack-v2.png)\n\nCloudflare has made a great dashboard for managing alerts.\nUnsee can aggregate alerts from multiple Alertmanager instances, running either in HA mode or separate.\nYou can access unsee at `http://:9094` using the admin user/password set via compose up:\n\n![Unsee](https://raw.githubusercontent.com/stefanprodan/swarmprom/master/grafana/screens/unsee.png)\n\n## Monitoring applications and backend services\n\nYou can extend swarmprom with special-purpose exporters for services like MongoDB, PostgreSQL, Kafka,\nRedis and also instrument your own applications using the Prometheus client libraries.\n\nIn order to scrape other services you need to attach those to the `mon_net` network so Prometheus\ncan reach them. Or you can attach the `mon_prometheus` service to the networks where your services are running.\n\nOnce your services are reachable by Prometheus you can add the dns name and port of those services to the\nPrometheus config using the `JOBS` environment variable:\n\n```yaml\n prometheus:\n image: stefanprodan/swarmprom-prometheus\n environment:\n - JOBS=mongo-exporter:9216 kafka-exporter:9216 redis-exporter:9216\n```\n\n","has_starred","hub_user","immutable_tags_settings",{"_74":11,"_75":76},"enabled","rules",[77],".*","is_automated","permissions",{"_81":21,"_82":11,"_83":11},"read","write","admin","user","storage_size",24076307,"orgData","recentTags",[90],{"_91":44,"_92":93,"_94":95,"_96":97,"_98":99,"_49":117,"_118":93,"_119":33,"_120":55,"_56":121,"_122":123,"_124":108,"_125":117,"_126":65,"_127":21},"content_type","creator",8489153,"full_size",12948287,"id",100092925,"images",[100],{"_101":102,"_103":104,"_105":106,"_107":108,"_109":110,"_111":112,"_113":106,"_114":-5,"_115":95,"_62":65,"_116":-5},"architecture","amd64","digest","sha256:b5c6063952e72bee5854b99b3fdf3a5a2244185218cbc669ab03069b66375d2d","features","","last_pulled","2026-10-05T19:48:13.488189755Z","last_pushed","2020-07-28T07:54:32Z","os","linux","os_features","os_version","size","variant","2020-07-28T07:54:31.789983Z","last_updater","last_updater_username","media_type","latest","repository",9029883,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":141,"_148":149,"_135":158,"_159":11},{"_40":142,"_42":143,"_45":46,"_47":35,"_48":11,"_49":50,"_51":52,"_53":144,"_56":35,"_32":33,"_57":58,"_59":44,"_60":61,"_62":63,"_64":65,"_66":-5,"_67":61,"_68":69,"_70":11,"_71":33,"_72":145,"_78":11,"_79":147,"_84":33,"_85":86},[],[44],[55],{"_74":11,"_75":146},[77],{"_81":21,"_82":11,"_83":11},"tags",{"_150":63,"_151":-5,"_152":-5,"_153":154},"count","next","previous","results",[155],{"_91":44,"_92":93,"_94":95,"_96":97,"_98":156,"_49":117,"_118":93,"_119":33,"_120":55,"_56":121,"_122":123,"_124":108,"_125":117,"_126":65,"_127":21},[157],{"_101":102,"_103":104,"_105":106,"_107":108,"_109":110,"_111":112,"_113":106,"_114":-5,"_115":95,"_62":65,"_116":-5},[],"canDeleteTags","actionData","errors"]

dhileepbalaji/node-exporter

Tags for dhileepbalaji/node-exporter