dfdsdk/metrics-datacollector repository overview

⁠metrics-datacollector

A generic metrics datacollector that scapes a provided open/prometheus metrics endpoint and convert the data to a standard JSON format and store in an AWS S3 bucket.

⁠Building the project

Run the following commands from the root directory to build the project:

mkdir -p bin
cd bin
go build ../cmd/data-collector
cd ..

⁠Building the Docker image

Run the following commands from the root directory to build the Docker image:

docker build -f build/package/Dockerfile -t datacollector:test .

⁠Running the application

We suggest running the application using the Docker image. The application requires the following environment variables to be set:

• DATA_COLLECTOR_METRICSENDPOINT: The endpoint to scrape the metrics from.
• DATA_COLLECTOR_S3BUCKETNAME: The name of the S3 bucket to store the collected data in.

The following environment variables are optional:

• DATA_COLLECTOR_SCHEDULE: How frequently in minutes to scrape the metrics endpoint. Defaults to 1440 minutes if not set.
• DATA_COLLECTOR_METRICSPREFIX: Prefix to add to the metrics names. Defaults to `` (no prefix) if not set.
• DATA_COLLECTOR_PROMPORT: Port number for exporting runtime metrics to Prometheus. Defaults to 2112 if not set.

⁠Prometheus metrics

This application exports the following runtime metrics for Prometheus:

data_collector_failed_runs
data_collector_remote_latency_ms
data_collector_timestamp
data_collector_successful_runs