dcroche/orbit repository overview

⁠Orbit

Kubernetes network load generator and measurement tool for validating network monitoring.

Orbit generates controlled traffic flows between pods (east-west) and external endpoints (north-south), independently measures them at application, wire, and system layers, and exposes all metrics via Prometheus — providing ground-truth data to validate tooling accuracy.

⁠Features

• Traffic Generation — TCP streams, UDP streams, HTTP requests, gRPC calls, ICMP pings, connection churn
• Three-Layer Measurement — Application-level byte/packet counters, wire-level TCP_INFO stats, system-level /proc metrics
• Peer Discovery — Automatic discovery via Kubernetes EndpointSlice API
• Leader Election — Kubernetes Lease-based election; leader coordinates traffic across all peers
• Scenario Engine — YAML-driven traffic profiles and active scenario selection loaded from ConfigMap with hot-reload via fsnotify
• Satellite Mode — Run an Orbit instance outside the cluster as a controlled external endpoint
• Authentication — Shared bearer token protecting all HTTP, gRPC, and raw TCP/UDP receiver endpoints
• Checksum Verification — SHA-256 payload integrity checks across HTTP and gRPC flows
• Prometheus Metrics — All measurements exposed at /metrics with a pre-built Grafana dashboard
• Helm Chart — DaemonSet or Deployment, RBAC, ServiceMonitor, PodDisruptionBudget, Satellite

⁠Quick Start

⁠Build

make build-local    # binary for current OS
make docker-build   # container image (current arch)
make docker-release # multi-arch image (amd64 + arm64) with SBOM & attestations

Run make help to see all available targets.

⁠Deploy with Helm

helm install orbit deploy/helm/orbit \
  --namespace orbit --create-namespace \
  --set auth.token="my-secret-token" \
  --set config.activeScenario="steady-low"

⁠Run Locally (development)

export ORBIT_AUTH_TOKEN=dev-token
export ORBIT_POD_NAME=local
./bin/orbit --mode=standalone --http-port=8080 --grpc-port=9090

⁠Version

./bin/orbit --version
# orbit version 0.1.0

The orbit_build_info Prometheus metric exposes version and commit labels at runtime.

⁠Architecture

graph TD
  subgraph agent["Orbit Agent"]
    direction TB
    Discovery
    Election
    Coordinator
    Scenario["Scenario Engine"]
    Generators
    Receivers
    Recorders
    HTTP_srv["HTTP Server :8080"]
    gRPC_srv["gRPC Server :9090"]
  end

  Election --> Coordinator
  Coordinator --> Scenario
  Scenario --> Generators
  Generators --> Recorders
  Receivers --> Recorders
  Recorders --> HTTP_srv
  gRPC_srv --> Coordinator
  Discovery --> Coordinator

⁠Data Flow

flowchart LR
  subgraph cluster["Kubernetes Cluster"]
    Leader(["Leader<br/><i>elected</i>"])
    Peer1(["Peer 1"])
    PeerN(["Peer N"])
  end

  Satellite(["Satellite<br/><i>external</i>"])
  Prom[("Prometheus")]

  Leader -- "gRPC schedule" --> Peer1
  Leader -- "gRPC schedule" --> PeerN
  Peer1 <-- "east-west<br/>TCP / UDP / HTTP / gRPC" --> PeerN
  Peer1 -- "north-south" --> Satellite
  Satellite -. "echo" .-> Peer1

  Prom -. "scrape /metrics" .-> Leader
  Prom -. "scrape /metrics" .-> Peer1
  Prom -. "scrape /metrics" .-> PeerN
  Prom -. "scrape /metrics" .-> Satellite

⁠Configuration

All flags can also be set via environment variable (prefix ORBIT_, uppercase, hyphens become underscores).

⁠Scenarios

Scenarios are defined in values.yaml under scenarios: and mounted as a ConfigMap. The file is watched via fsnotify — changes to both scenario definitions and the active scenario are picked up automatically without restarting pods.

The active scenario is set via config.activeScenario in your Helm values. To switch scenarios at runtime:

helm upgrade orbit deploy/helm/orbit --reuse-values \
  --set config.activeScenario="connection-churn"

Kubernetes propagates the ConfigMap update to all pods (~60s), and the leader automatically stops existing flows and activates the new scenario.

⁠Schedule Lease

The leader periodically heartbeats the current schedule to all peers (default every 10s). Each peer tracks the last heartbeat time and enforces a lease TTL (default 30s). If a peer loses contact with the leader — due to a network partition, leader crash, or leadership change — its lease expires and generators are stopped automatically. This prevents orphaned traffic flows from running indefinitely.

helm upgrade orbit deploy/helm/orbit --reuse-values \
  --set config.scheduleLeaseTTL="30s" \
  --set config.heartbeatInterval="10s"

Heartbeats are idempotent: peers recognize repeated schedules by runID and refresh their lease without restarting generators.

Before distributing schedules, the leader waits for the peer mesh to stabilize — the discovered peer count must remain unchanged for config.stabilizationPeriod (default 10s). This prevents partial mesh assignments when pods are still joining. Adjust it for larger clusters:

helm upgrade orbit deploy/helm/orbit --reuse-values \
  --set config.stabilizationPeriod="30s"

scenarios:
  steady-low:
    description: "Low sustained load"
    eastWest:
      - type: tcp-stream
        bandwidthMbps: 10
        payloadBytes: 1400
      - type: http
        rps: 10
        payloadBytes: 512
    northSouth: []

  connection-churn:
    description: "Rapid connection lifecycle"
    eastWest:
      - type: connection-churn
        connectionsPerSecond: 500
        holdDurationMs: 50
    northSouth: []

⁠Flow Types

⁠Satellites

Satellites are external Orbit instances (typically running via Docker Compose) that act as controlled north-south endpoints. Register them under northSouth.satellites in your Helm values — you only need to specify the host once and the target URLs are resolved automatically from well-known ports:

northSouth:
  satellites:
    - name: satellite-01
      host: "10.0.0.50"
      # ports default to 8080/9090/10000/11000 — override if needed
      # authToken: ""    # defaults to the cluster's auth.token
      flows:
        - type: http
          rps: 50
          payloadBytes: 1024
        - type: tcp-stream
          bandwidthMbps: 10
          payloadBytes: 1400

Satellite flows are started automatically whenever any scenario is activated. They are merged with the scenario's own northSouth flows and appear in Prometheus metrics with direction="north-south". Each satellite can optionally override the cluster auth token via authToken.

To collect the satellite's own metrics (receiver-side bytes, active connections, checksum errors), enable serviceMonitor.satelliteServiceMonitor.enabled — see Helm Values⁠ below.

To run a satellite outside the cluster, see deploy/compose/README.md.

⁠Endpoints

⁠Prometheus Metrics

⁠Build Info

⁠Cluster

⁠Application Layer

⁠Wire Layer (Linux only, TCP_INFO)

Wire-layer byte and segment counters require Linux kernel 4.2+ for the extended tcp_info fields (bytes_sent, bytes_received, bytes_retrans, segs_out). On older kernels these counters report zero but all other TCP_INFO metrics (RTT, cwnd, MSS, etc.) still work.

⁠System Layer (Linux only, /proc)

⁠Generator Metrics

⁠Receiver Metrics

⁠Observability

A pre-built Grafana dashboard is available at deploy/grafana/orbit-dashboard.json. Import it and select a Prometheus datasource.

Prometheus recording rules and alerting rules are at deploy/prometheus/recording-rules.yaml. Included alerts:

⁠Helm Values

See deploy/helm/orbit/values.yaml for all configurable values. Key options:

• mode — daemonset (one per node) or deployment (replica count)
• auth.token / auth.existingSecret — Bearer token configuration
• config.activeScenario — Active scenario (set in ConfigMap, hot-reloaded without restart)
• config.stabilizationPeriod — Time the peer mesh must be stable before distributing schedules (default 10s)
• config.scheduleLeaseTTL — Peer stops generators if no leader heartbeat received within this duration (default 30s)
• config.heartbeatInterval — Leader sends schedule heartbeats to peers at this interval (default 10s)
• northSouth.satellites — Register external satellite endpoints with their traffic flows (hot-reloaded)
• serviceMonitor.enabled — Create Prometheus ServiceMonitor for orbit pods
• serviceMonitor.satelliteServiceMonitor.enabled — Create ServiceMonitor + headless Service + Endpoints for external satellites (IPs sourced from northSouth.satellites[].host). Supports labels and annotations for Prometheus Operator discovery
• satellite.enabled — Deploy a satellite instance inside the cluster
• securityContext.capabilities.add: [NET_RAW] — Required for ICMP

⁠Make Targets

⁠Project Structure

orbit/
├── cmd/orbit/main.go              # Entrypoint, version/commit injection
├── internal/
│   ├── agent/                      # Agent orchestration, mode dispatch
│   ├── auth/                       # Token validation, HTTP/gRPC middleware
│   ├── checksum/                   # SHA-256 payload integrity verification
│   ├── config/                     # Configuration loading (flags, env, file)
│   ├── coordinator/                # Leader → peer schedule distribution
│   ├── discovery/                  # Peer discovery via headless service
│   ├── election/                   # Kubernetes Lease-based leader election
│   ├── generator/                  # Traffic generators (TCP, UDP, HTTP, gRPC, ICMP, Churn)
│   ├── metrics/                    # Prometheus metric definitions
│   ├── receiver/                   # Traffic receivers (TCP, UDP, HTTP, gRPC)
│   ├── recorder/                   # Measurement recorders (App, Wire, System)
│   ├── scenario/                   # Scenario engine + fsnotify config watcher
│   └── server/                     # HTTP & gRPC servers
├── proto/orbit/v1/                 # Protobuf service & message definitions
├── deploy/
│   ├── compose/                    # Docker Compose for external satellite
│   ├── helm/orbit/                 # Helm chart (DaemonSet, Deployment, Satellite)
│   ├── grafana/                    # Grafana dashboard JSON
│   └── prometheus/                 # Recording rules & alerting rules
├── Dockerfile                      # Multi-arch build (amd64 + arm64)
├── Makefile                        # Build, test, release targets
├── VERSION                         # Semantic version (read by Makefile)
├── LICENSE                         # Apache-2.0
└── go.mod

⁠Operational Notes

⁠Prometheus Cardinality (run_id label)

Several high-cardinality metrics (orbit_app_bytes_sent_total, orbit_app_bytes_received_total, orbit_app_connections_total, etc.) carry a run_id label. Each scenario activation generates a new timestamp-based run_id (e.g. steady-low-1712345678901), creating a permanently new set of Prometheus timeseries.

In long-running deployments with frequent scenario activations, this causes unbounded timeseries growth in the Prometheus TSDB, which can eventually lead to OOM or degraded query performance.

Recommended mitigations:

• Set a short TSDB retention window — --storage.tsdb.retention.time=7d (or match your alerting lookback window). Old run_id timeseries will be evicted after retention expires.
• Tune --query.max-samples — The default (50000000) may be hit by range queries over many run_id values. Raise or lower based on available RAM.
• Limit scenario churn — Avoid activating new scenarios more often than needed. Each activation creates a new run_id.
• Use recording rules — Pre-aggregate high-cardinality series in deploy/prometheus/recording-rules.yaml to reduce query-time cardinality.

⁠License

Apache License 2.0⁠