Sign inSign up

statsbomb/prometheus-msk-discovery

By statsbomb

•Updated over 1 year ago

Discovers AWS MSK clusters and outputs a Prometheus static config for use with file_sd_config

Image
0

100K+

statsbomb/prometheus-msk-discovery repository overview

statsbomb/prometheus-msk-discovery Test Go Report Card

⁠prometheus-msk-discovery

Service discovery for AWS MSK⁠, compatible with Prometheus⁠.

⁠How it works

This service gets a list of MSK clusters in an AWS account and exports each broker to a Prometheus-compatible static config to be used with the file_sd_config⁠ mechanism.

⁠Pre-requisites

⁠1) IAM Policy

When using AWS credentials or IAM Roles, the following policy needs to be attached to the role/user being used:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka:ListClusters",
                "kafka:ListNodes"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
⁠2) AWS Credentials
  • No special provisions are made to obtain AWS credentials and instead that process is left to the AWS SDK. Credentials will be searched for automatically in the order specified in the documentation⁠.

⁠Running it

Usage of ./prometheus-msk-discovery:
  -job-prefix string
    	string with which to prefix each job label (default "msk")
  -output string
    	path of the file to write MSK discovery information to (default "msk_file_sd.yml")
  -scrape-interval duration
    	interval at which to scrape the AWS API for MSK cluster information (default 5m0s)
⁠Example output:
$ ./prometheus-msk-discovery -scrape-interval 10s
2021/04/04 21:02:55 Writing 1 discovered exporters to msk_file_sd.yml

An example output file can be found here⁠

⁠Integration with kube-prometheus-stack

The Docker image⁠ for this project can be used to inject a container into the Prometheus Spec of a kube-prometheus-stack⁠ installation by using the following snippet in values.yaml:

prometheus:
  prometheusSpec:
    containers:
      - name: prometheus-msk-discovery
        image: statsbomb/prometheus-msk-discovery:latest
        args:
          - -output
          - /config-out/msk_file_sd.yml
        volumeMounts:
          - name: config-out
            mountPath: /config-out

You'll then need to add something similar to the following to your additionalScrapeConfig:

- job_name: prometheus-msk-discovery
  file_sd_configs:
    - files:
        - /etc/prometheus/config_out/msk_file_sd.yml
  honor_labels: true

⁠Building

Building can be done just by using go build

⁠Contributing

Pull requests, issues and suggestions are appreciated.

⁠Credits

⁠License

Apache License 2.0, see LICENSE⁠.

Tag summary

Content type

Image

Digest

sha256:48eea7729…

Size

10.2 MB

Last updated

over 1 year ago

docker pull statsbomb/prometheus-msk-discovery