Sign inSign up

talkopsai/tempo-mcp-server

By talkopsai

•Updated 3 months ago

MCP Server for Grafana Tempo trace search, analysis, RED metrics, and cross-pillar correlation

Image
Developer tools
Monitoring & observability
0

161

talkopsai/tempo-mcp-server repository overview

⁠Tempo MCP Server by TalkOps.ai⁠

An MCP server that gives AI assistants the power to search, analyze, summarize, and correlate distributed traces from Grafana Tempo — with TraceQL query construction, RED metrics analysis, cross-pillar pivots, service topology mapping, and operational diagnostics.

⁠✨ Features

  • Smart Trace Search: Translate K8s-friendly filters (namespace, service, deployment) into valid TraceQL, enforce query guardrails (time ranges, limits), and return compact trace summaries.
  • Intelligent Trace Analysis: Fetch a trace, extract the critical path, identify error spans, detect the suspected root cause, and recommend follow-up queries — all in a single tool call.
  • RED Metrics Triage: Execute TraceQL metrics queries (rate(), quantile_over_time(), count_over_time()) and pivot from aggregated metrics to concrete traces via exemplars.
  • Cross-Pillar Correlation: Extract trace IDs from log lines and retrieve full traces. Pivot from metrics spikes to exemplar traces. Correlate related traces using multiple strategies.
  • Backend Diagnostics: Aggregate health checks, build info, component service status, and ring member health into severity-ranked diagnostics with remediation steps.
  • Service Topology: Map service dependencies from Tempo's metrics-generator service graph data, with request rates and error rates per edge.
  • Multi-Tenancy: Per-backend tenant header injection (X-Scope-OrgID), cross-tenant queries, and tenant ID validation.
  • Tempo Operator CRD Management: List, inspect, create, and patch TempoStack and TempoMonolithic custom resources with dry-run safety.

⁠🚀 How to Use This Image

This Docker image runs an MCP server that connects to your Grafana Tempo backend(s). AI assistants (Claude, Cline, Cursor, or any MCP client) connect to the server over HTTP, SSE, or stdio.

⁠Running Standalone (HTTP Transport)
docker run --rm -it \
  -p 8768:8768 \
  -e TEMPO_BASE_URL=http://host.docker.internal:3200 \
  -e MCP_TRANSPORT=http \
  talkopsai/tempo-mcp-server:latest

The server is now listening on http://localhost:8768/mcp.

Then configure your MCP client to connect:

{
  "mcpServers": {
    "tempo": {
      "url": "http://localhost:8768/mcp",
      "description": "MCP Server for Grafana Tempo distributed tracing"
    }
  }
}
⁠With Kubernetes Discovery

To enable auto-discovery of Tempo backends via Kubernetes service labels and Tempo Operator CRDs:

docker run --rm -it \
  -p 8768:8768 \
  -v ~/.kube:/app/.kube:ro \
  -e TEMPO_BASE_URL=http://host.docker.internal:3200 \
  -e K8S_KUBECONFIG=/app/.kube/config \
  -e K8S_ENABLED=true \
  -e MCP_TRANSPORT=http \
  talkopsai/tempo-mcp-server:latest

Tip: Mount the full ~/.kube directory (not just config) so certificate paths referenced in your kubeconfig are available inside the container.

⁠Multi-Backend Setup

Connect to multiple Tempo backends simultaneously:

docker run --rm -it \
  -p 8768:8768 \
  -e MCP_TRANSPORT=http \
  -e TEMPO_BACKENDS='[
    {"id": "prod", "base_url": "https://tempo-prod.example.com", "type": "tempo", "multi_tenant": true, "default_tenant": "team-a"},
    {"id": "staging", "base_url": "http://tempo-staging:3200", "type": "tempo"}
  ]' \
  talkopsai/tempo-mcp-server:latest

⁠🛠️ Configuration (Claude Desktop — Stdio)

To use this image over stdio transport directly inside Claude Desktop:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "tempo-mcp-server": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e", "TEMPO_BASE_URL=http://host.docker.internal:3200",
        "-e", "MCP_TRANSPORT=stdio",
        "-e", "MCP_LOG_LEVEL=INFO",
        "talkopsai/tempo-mcp-server:latest"
      ]
    }
  }
}

For multi-tenant backends, add tenant environment variables:

{
  "mcpServers": {
    "tempo-mcp-server": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e", "TEMPO_BASE_URL=https://tempo.example.com",
        "-e", "TEMPO_MULTI_TENANT=true",
        "-e", "TEMPO_DEFAULT_TENANT=my-team",
        "-e", "TEMPO_AUTH_HEADER=Bearer your-token",
        "-e", "MCP_TRANSPORT=stdio",
        "-e", "MCP_LOG_LEVEL=INFO",
        "talkopsai/tempo-mcp-server:latest"
      ]
    }
  }
}

⁠⚙️ Environment Variables

⁠Server Configuration
VariableDefaultDescription
MCP_TRANSPORTstdioTransport protocol: http, sse, streamable-http, or stdio
MCP_HOST0.0.0.0Host interface to bind to
MCP_PORT8768Port for HTTP server
MCP_PATH/mcpMCP endpoint path
MCP_LOG_LEVELINFOLog level: DEBUG, INFO, WARNING, ERROR
MCP_LOG_FORMATjsonLog format: json or text
MCP_HTTP_TIMEOUT300HTTP server timeout (seconds)
⁠Tempo Backend (Single)
VariableDefaultDescription
TEMPO_BASE_URLhttp://localhost:3200Tempo HTTP API base URL
TEMPO_BACKEND_IDdefaultBackend identifier
TEMPO_TYPEtempoBackend type: tempo, tempo-gateway, unknown
TEMPO_DEPLOYMENT_MODEunknownDeployment mode: monolithic, microservices, unknown
TEMPO_AUTH_HEADER(empty)Authorization header (e.g. Bearer <token>)
TEMPO_VERIFY_SSLtrueVerify SSL certificates
TEMPO_TIMEOUT30HTTP timeout per request (seconds)
TEMPO_BACKENDS(empty)JSON array of backend configs for multi-backend mode
⁠Multi-Tenancy
VariableDefaultDescription
TEMPO_MULTI_TENANTfalseEnable multi-tenant mode
TEMPO_DEFAULT_TENANT(empty)Default tenant ID (required if multi-tenant)
TEMPO_TENANT_HEADERX-Scope-OrgIDHTTP header for tenant ID injection
⁠Query Guardrails
VariableDefaultDescription
TEMPO_MAX_LOOKBACK168hMaximum query lookback (7 days)
TEMPO_DEFAULT_SEARCH_LIMIT20Default max traces per search
TEMPO_MAX_SEARCH_LIMIT100Absolute max traces per search
TEMPO_REQUIRE_TIME_RANGEtrueRequire time range on searches
TEMPO_REQUIRE_FILTER_OR_QUERYtrueRequire at least one filter or TraceQL query
TEMPO_MAX_METRICS_DURATION3hMaximum metrics query time range
⁠LLM Format
VariableDefaultDescription
TEMPO_LLM_FORMATtrueEnable LLM-optimized trace format (Tempo 2.9+)
⁠Kubernetes Discovery
VariableDefaultDescription
K8S_ENABLEDfalseEnable K8s-based Tempo backend discovery
K8S_CONTEXT(empty)Specific kubeconfig context to use
K8S_IN_CLUSTERfalseSet true when running inside a Kubernetes pod

⁠🔧 Available Tools (23 Total)

⁠Discovery
ToolDescription
tempo_list_backendsList all configured Tempo backends with health status
tempo_get_backendGet detailed backend profile: health, version, capabilities, tenant requirements
tempo_get_query_policiesGet query guardrails and default search parameters
⁠Schema Discovery
ToolDescription
tempo_get_attribute_namesDiscover trace attribute names by scope (resource, span, intrinsic, event, link)
tempo_get_attribute_valuesGet distinct values for a specific trace attribute
tempo_get_k8s_attribute_mapGet canonical K8s-to-Tempo attribute mapping with optional live validation
⁠Search & Retrieval
ToolDescription
tempo_traceql_searchHIGH-INTENT: Search using raw TraceQL or K8s-friendly filters with auto-translation
tempo_get_traceRetrieve a single trace by ID with LLM-optimized format
tempo_query_a2uiRetrieve a trace structured for A2UI rendering with DAG-aware pruning
tempo_summarize_traceHIGH-INTENT: Generate intelligent trace summary — critical path, root cause, next queries
tempo_find_related_tracesHIGH-INTENT: Find related traces using correlation strategies
⁠Metrics
ToolDescription
tempo_traceql_metrics_rangeTraceQL metrics range query — Prometheus-compatible time series output
tempo_traceql_metrics_instantTraceQL metrics instant query — point-in-time vector output
⁠Cross-Pillar Pivots
ToolDescription
tempo_get_exemplar_tracesPivot from aggregated metrics to concrete traces via exemplars
tempo_get_trace_from_logExtract trace ID from a log line and retrieve the full trace
⁠Diagnostics
ToolDescription
tempo_get_diagnosticsHIGH-INTENT: Comprehensive backend diagnostics with severity-ranked findings
⁠Topology
ToolDescription
tempo_get_service_dependenciesMap service dependencies with request rates per edge
⁠Operator CRD Management
ToolDescription
tempo_list_operator_crsList Tempo Operator custom resources across namespaces
tempo_get_operator_crGet Tempo Operator CR with full spec, status, and storage config
tempo_create_operator_crCreate TempoStack or TempoMonolithic CR (dry_run by default)
tempo_patch_operator_crPatch fields of an existing Tempo Operator CR (dry_run by default)
⁠Trace Comparison
ToolDescription
tempo_compare_tracesHIGH-INTENT: 5-dimensional diff of two traces
⁠Alerting
ToolDescription
tempo_generate_alerting_expressionGenerate PromQL alerting expressions from trace patterns

⁠📖 Available Resources (11 URIs)

ResourceDescription
tempo://system/backendsAll configured backends with health status — start here
tempo://system/backends/{backend_id}Detailed profile for a specific backend
tempo://deployment/overviewDeployment topology: backends, modes, tenants, K8s status
tempo://reference/traceqlTraceQL syntax reference and examples
tempo://reference/traceql-metricsTraceQL metrics functions reference
tempo://reference/k8s-attributesCanonical K8s-to-Tempo attribute mapping
tempo://reference/query-policiesQuery guardrails and safety guidelines
tempo://runbooks/latency-spikeRunbook for investigating latency spikes
tempo://runbooks/error-burstRunbook for investigating error bursts
tempo://runbooks/no-traces-foundDiagnostic runbook for "no traces found"
tempo://examples/common-queriesCommon TraceQL and metrics query examples

⁠🔐 Security Best Practices

  1. Never expose the MCP server to the public internet without proper authentication.
  2. All tools are read-only — the server only performs HTTP GET requests against Tempo's query APIs. No data is created, modified, or deleted.
  3. Read-Only Mount: When using K8s discovery, always mount kubeconfig with :ro.
  4. Tenant isolation: In multi-tenant deployments, verify tenant IDs are correctly scoped to prevent cross-tenant data leakage.
  5. Protect auth headers: TEMPO_AUTH_HEADER is included in every request — treat it as a secret.
  6. Query guardrails: The server enforces time range, limit, and filter requirements. Review and tune for your environment.
  7. Operator CRD tools: tempo_create_operator_cr and tempo_patch_operator_cr default to dry_run=True. Review output before applying with dry_run=False.

⁠🐳 Docker Compose Example

services:
  tempo-mcp-server:
    image: talkopsai/tempo-mcp-server:latest
    ports:
      - "8768:8768"
    environment:
      - TEMPO_BASE_URL=http://tempo:3200
      - MCP_TRANSPORT=http
      - MCP_LOG_LEVEL=INFO
    depends_on:
      - tempo

  tempo:
    image: grafana/tempo:latest
    ports:
      - "3200:3200"
    command: -config.file=/etc/tempo/tempo.yaml
    volumes:
      - ./tempo.yaml:/etc/tempo/tempo.yaml

If you find this MCP server useful, consider leaving a ⭐ on the GitHub repository⁠!

Tag summary

Content type

Image

Digest

sha256:5880cb1a5…

Size

82.1 MB

Last updated

3 months ago

docker pull talkopsai/tempo-mcp-server