prometheus_aggregator receives prometheus style samples, aggregates them and than exposes as prometheus metrics.
Project is in very early stage. It's tested but with minimal functionality. There is almost no optimisation done yet and a few known unbound memory allocations.
Short-lived client is shooting samples via UDP toward aggregator server which parses, aggregates and stores them in memory.
The storage is then scraped using standard Prometheus HTTP endpoint (both text and binary exposition formats are supported).
+----------+ +-------------------------+ +--------------+ | client |---(UDP)--->| prometheus_aggregator |<---(scrape /metrics)---| Prometheus | +----------+ +-------------------------+ +--------------+
Ingress format for samples is a text, line based format with two types of lines:
- shared labels
Each line should be terminated with single new-line.
shared labels line
List of labels shared between all samples in the packet. Designed to lower the packet size by removing duplicates.
If present, it must be first line of the packet.
There is only one shared labels line allowed per packet.
service=srvA1;host=hostA;phpVersion=5.6 name_of_1_metric_total|c|labelA=labelValueA;label2=labelValue2|12.345 name_of_2_metric_total|c|56 name_of_3_metric|g|7.3
|name||name of the metric||a-zA-Z0-9_|
|type||type of the metric||counter: c<br>gauge: g<br>histogram with linear buckets: hl|
|type config||additional configuration for the type<br>currently used only for histograms|
|labels||pairs of name and value separated by semicolon (;)<br>field is optional||name: a-zA-Z0-9<br>value: a-zA-Z0-9.|
|value||sample value<br>negative values are not yet supported||0-9.|
As of now following metrics are supported:
- histogram with linear buckets
Histograms with linear buckets
Type config values are passed to LinearBuckets(start, width float64, count int)
There are two major components: sample server and collector.
Sample server is responsible for listening for the incoming samples via UDP, parsing each packet to samples and handing over to collector for processing.
As of now there is single goroutine responsible for reading and parsing.
Collector is responsible for:
- processing of the samples to metrics metrics,
- storing metrics in memory,
- exposing metrics for scraping.
Collector implements prometheus.Collector interface.
New samples are buffered in ingress channel and then picked-up by a processor, converted to metrics and stored.
Processor is implemented as single goroutine.
|app_start_timestamp_seconds||collector||gauge||second||Unix timestamp of the app collector start.|
|app_duration_seconds||collector||gauge||second||Time in seconds since start of the app.|
|app_collector_queue_length||collector||gauge||-||Number of elements waiting in collector queue for processing.|
|app_collector_processing_duration_ns||collector||summary||nanosecond||Duration of the processing in the collector in ns.|
|app_ingress_requests_total||server||counter||-||Number of request entering server.|
|app_ingress_samples_total||server||counter||-||Number of samples entering server.|
|app_ingress_request_handling_duration_ns||server||summary||nanosecond||Time in ns spent on handling single request.|
govend is used for vendoring.
govend -v go build
// UdpHost is address on which UDP server is listening UDPHost string `envconfig:"default=0.0.0.0"` // UdpPort is port number on which UDP server is listening UDPPort int `envconfig:"default=8080"` // UDPBufferSize is a size of a buffer in bytes used for incoming samples. // Sample not fitting in buffer will be partially discarded. // Sync buffer size with client. UDPBufferSize int `envconfig:"default=4096"` // MetricsHost is address on which metric server for prometheus is listening MetricsHost string `envconfig:"default=0.0.0.0"` // MetricsHost is port number on which metric server for prometheus is listening MetricsPort int `envconfig:"default=9090"` // LogLevel is a minimal log severity required for the message to be logged. // Valid levels: [debug, info, warn, error, fatal, panic]. LogLevel string `envconfig:"default=info"`
# !/usr/bin/env bash export APP_UDP_HOST="0.0.0.0" export APP_UDP_PORT="9090" export APP_UDP_BUFFER_SIZE="2048" export APP_METRICS_HOST="0.0.0.0" export APP_METRICS_PORT="8080" export APP_LOG_LEVEL="DEBUG" ./prometheus-aggregator
$ go test
Dedicated tests for race detection:
$ go test ./ -run Test_Race_ -race -count 1000 -cpu 1,2,4,8,16
docker build -t prometheus-aggregator . docker run -it --rm -p 10901:8080 -p 10902:9090 --name prometheus_aggregator prometheus-aggregator