apptality/aws-ecs-cloudmap-prometheus-discovery repository overview

⁠AWS ECS & CloudMap Prometheus Discovery

⁠Overview

This application⁠ facilitates discovery of ECS and/or CloudMap resources, specifically tailored to be compatible with Prometheus HTTP Service Discovery⁠.

This solution leverages AWS API to dynamically discover:

• ECS Clusters⁠, related ECS Services, and ECS Tasks
• CloudMap Namespaces⁠, registered Service Connect and Service Discovery services, and their instances

making it easier for Prometheus to monitor these services without hardcoding their addresses or ports, or using any kind of file-based service discovery.

⁠Features

Below is a high level overview of what application is capable of. For the full list of supported configuration parameters and how to override these using Docker Environment Variables⁠ please refer to appsettings.json⁠

• Prometheus Compatibility: By default, exposes 9001:/prometheus-targets endpoint which response is compatible with <http_sd_config>⁠ of Prometheus configuration file, as well as with OpenTelemetry Prometheus Receiver⁠ (see example below).

• ECS Discovery: Supports discovery of ECS clusters, services, running tasks

  • Supply EcsClusters as semicolon separated string of ECS clusters to include: "cluster1;cluster2;"
  • Supports filtering of ECS Services that should be included in the Prometheus response based on Resource Tags (see EcsServiceSelectorTags configuration property)
  • Supports filtering to control which tags should be included as labels in the response (see EcsTaskTags, EcsServiceTags configuration properties)

• AWS CloudMap Integration: Supports discovery of CloudMap namespaces, services and instances

  • Supply CloudMapNamespaces as semicolon separated string of CloudMap namespaces to include: "namespace1;"
  • Supports filtering of CloudMap Services that should be included in the Prometheus response based on Resource Tags (see CloudMapServiceSelectorTags configuration property)
  • Supports filtering to control which tags should be included as labels in the response (see CloudMapServiceTags, CloudMapNamespaceTags configuration properties)

• Scalable and Flexible: Built to be scalable and flexible, adapting to changes in the service infrastructure

  • Allows having any number of port-metrics pairs per each running ECS Task (useful when your ECS Task Definition defines multiple containers, which also expose ports and metrics endpoint). See MetricsPathPortTagPrefix configuration property for more details.
  • Allows supplying static set of labels to be added to every Prometheus target (see ExtraPrometheusLabels configuration property)
  • Supports rich mechanism for re-labling, so you don't have to fix your Grafana dashboards. See RelabelConfigurations configuration property for more details.
  • Exposes health check endpoint: /health

At least one of EcsClusters or CloudMapNamespaces must be provided for application to work. If you would like to include both Service Connect⁠ and Service Discovery⁠ targets, CloudMapNamespaces must be supplied (with or without EcsClusters). When both parameters are supplied, the end result is only an intersection of targets that exist in ECS clusters supplied and CloudMap namespaces provided.