kanewinter/metrics-server-prom-aws repository overview

⁠Kubernetes Metrics Server Prometheus Adapter

Motivation⁠ | Usage⁠ | Docker Compose⁠ | Metrics transformation⁠ | License⁠

⁠Motivation

⁠What is provided

A Docker image on which Prometheus⁠ can scrape Kubernetes metrics provided by metrics-server⁠. The image can run anywhere where Prometheus can use it as a target, even in Kubernetes itself.

⁠Why is it needed

metrics-server seems to be the successor of heapster⁠ for Kubernetes monitoring. However, metrics-server currently only provides its metrics in JSON format via the Kubernetes API server.

Prometheus on the other hand expects a special text-based format⁠. So in order for Prometheus to scrape those metrics, they must be transparently transformed from JSON to its own format on every request.

⁠Differences

Other than metrics-server itself, this Docker container provides additional metrics metadata that are retrieved via kubectl API calls and included in the Prometheus output.

⁠How does it work

The following diagram illustrates how the format transformation is achieved:

1. Prometheus scrapes the Docker container on :9100/metrics
2. Inside the Docker container uwsgi⁠ is proxying the request to kube proxy⁠
3. kube-proxy reads the provided config file and tunnels the request into Kubernetes to metrics-server
4. metrics-server replies with JSON formatted metrics
5. kube proxy forwards the request back to uwsgi
6. uwsgi calls transform.py⁠
7. transform.py rewrites the JSON into Prometheus readable output and hands the result back to uwsgi
8. uwsgi sends the final response back to Prometheus

⁠Usage

⁠Run metrics-server-prom

Simply run the Docker image with a kube config mounted into /etc/kube/config.

$ docker run -d \
    -p 9100:9100 \
    -v ${HOME}/.kube/config:/etc/kube/config:ro \
    cytopia/metrics-server-prom

If your kube config contains multiple contexts, you can tell metrics-server-prom what context to use, to connect to the cluster.

$ docker run -d \
    -p 9100:9100 \
    -v ${HOME}/.kube/config:/etc/kube/config:ro \
    -e KUBE_CONTEXT=my-context \
    cytopia/metrics-server-prom

⁠Configure Prometheus

prometheus.yml:

scrape_configs:
  - job_name: 'kubernetes'
    scrape_interval: '15s'
    metrics_path: '/metrics'
    static_configs:
      - targets:
        - <DOCKER_IP_ADDRESS>:9100

⁠Docker Compose

To test this out locally, this repository ships an example Docker Compose setup with metrics-server-prom and Prometheus.

1. Navigate to example/⁠
2. Copy env-example to .env
3. Adjust KUBE_CONTEXT in .env
4. Run it docker-compose up

⁠Metrics transformation

metrics-server provices metrics in the following format:

{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
  },
  "items": [
    {
      "metadata": {
        "name": "etcd-server-events-abc",
        "namespace": "kube-system",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/etcd-server-events-ip-10-30-78-99.eu-central-1.compute.internal",
        "creationTimestamp": "2018-08-20T03:19:05Z"
      },
      "timestamp": "2018-08-20T03:19:00Z",
      "window": "1m0s",
      "containers": [
        {
          "name": "etcd-container",
          "usage": {
            "cpu": "7m",
            "memory": "125448Ki"
          }
        }
      ]
    },

  ]
}

metrics-server-prom transforms it to the following format:

Note: Additional metadata (node and ip) have been added.

# HELP kube_metrics_server_pod_cpu The CPU time of a pod in seconds.
# TYPE kube_metrics_server_pod_cpu gauge
kube_metrics_server_pod_cpu{node="ip-10-30-78-99.eu-central-1.compute.internal",pod="etcd-server-events-abc",ip="10.30.62.138",container="etcd-container",namespace="kube-system"} 420
# HELP kube_metrics_server_pod_mem The memory of a pod in KiloBytes.
# TYPE kube_metrics_server_pod_mem gauge
kube_metrics_server_pod_mem{node="ip-10-30-78-99.eu-central-1.compute.internal",pod="etcd-server-events-abc",ip="10.30.62.138",container="etcd-container",namespace="kube-system"} 128475136

⁠License

MIT License⁠

Copyright (c) 2018 cytopia⁠