lanedfritz/vllm-dashboard repository overview

⁠vLLM Dashboard

A self-hosted dashboard for monitoring and controlling one or more vLLM⁠ inference servers on your own network — live metrics, historical usage, model hot-swapping, and a transparent request proxy with automatic model swapping, idle GPU unloading, and startup preloading.

Think of it as llama-swap, but with a UI: a stable OpenAI-compatible endpoint that your tools (OpenWebUI, opencode, any OpenAI SDK client) talk to directly, which swaps the backend container to the right model on demand, unloads it when idle to free the GPU, and brings it back automatically when the dashboard restarts.

⁠How this is meant to be used

There are two usage tiers, and it matters which one applies to you:

• Monitoring + transparent proxy (no Docker required on the backend) — if your vLLM server is already running and reachable over HTTP, you can add it as a source and get live metrics, history, and the /proxy/<source>/v1 endpoint for free. This works whether that server runs bare-metal, in a systemd service, in a venv, or in a container you don't want this app touching.
• Hot-swap, idle-unload, startup preload, auto-swap-on-request — these all work by literally stopping and recreating a Docker container (a new model means a new docker run with different flags), so they only work when the backend itself runs as a Docker container on a host whose Docker Engine API this dashboard can reach. There's no equivalent for a bare-metal process — the dashboard has nothing to stop/restart in that case. See "Hot-swap requirements" below for what that needs. If you don't containerize vLLM, the monitoring tier above is all you get, and that's a perfectly legitimate way to use this app.

⁠Features

• Live monitoring — requests running/waiting, KV cache usage, throughput, latency (TTFT/ITL/E2E), scraped straight from vLLM's Prometheus /metrics, with 30-day history and hourly rollups.
• Model-driven hot-swap — a curated, typed catalog of vLLM launch options (dropdowns and suggestions instead of free-text flags) so swapping models doesn't mean guessing CLI syntax.
• Transparent proxy with auto-swap — point any OpenAI-compatible client at /proxy/<source>/v1; the dashboard swaps the backend container to the requested model automatically before forwarding, and answers /v1/models with the full registered catalog, not just whatever happens to be loaded.
• Idle unload + startup preload — stop a model's container automatically after it sits idle past a configurable TTL to free the GPU, and bring it back up on its own when the dashboard restarts if it isn't already running.
• Live traffic view — watch the messages sent, the model's reasoning, and its response stream in through the proxy in real time — actually useful for debugging why a model answered the way it did, not just a request counter.
• Live container logs — stream a backend container's stdout/stderr in the UI instead of docker logs -f-ing into a terminal.
• Multi-instance — any number of sources (vLLM or Ollama), each with its own metrics, model registry, and launch settings.
• Docker-native — hot-swap talks straight to the Docker Engine API and keeps a swapped container's Compose stack labels intact, so it doesn't drop out of Portainer's stack view.

⁠Quick start

docker run -d \
  --name vllm-dashboard \
  -p 8080:8080 \
  -e ConnectionStrings__DefaultConnection="Host=<your-postgres-host>;Port=5432;Database=dashboard;Username=dashboard;Password=dashboard" \
  -e ASPNETCORE_ENVIRONMENT=Production \
  -e DATAPROTECTION_KEYS_DIR=/data/dataprotection \
  -v vllm-dashboard-data:/data \
  lanedfritz/vllm-dashboard:latest

You'll need a PostgreSQL database reachable from the container — see the Compose example below for a complete self-contained setup, including Postgres.

Open http://<host>:8080, register the first account (it becomes the admin automatically), and add your first source from the Settings page.

⁠Docker Compose

services:
  postgres:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_DB: dashboard
      POSTGRES_USER: dashboard
      POSTGRES_PASSWORD: dashboard
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dashboard -d dashboard"]
      interval: 5s
      timeout: 5s
      retries: 10
    expose:
      - "5432"

  dashboard:
    image: lanedfritz/vllm-dashboard:latest
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      ConnectionStrings__DefaultConnection: "Host=postgres;Port=5432;Database=dashboard;Username=dashboard;Password=dashboard"
      ASPNETCORE_ENVIRONMENT: Production
      ASPNETCORE_URLS: "http://+:8080"
      DATAPROTECTION_KEYS_DIR: /data/dataprotection
      # Only needed for a source you want to hot-swap models on, e.g. tcp://<servername-or-ip>:2375
      DOCKER_HOST: ""
    ports:
      - "8080:8080"
    volumes:
      - dataprotection:/data

volumes:
  pgdata:
  dataprotection:

Change POSTGRES_PASSWORD (and the matching password in the connection string) before running this anywhere other than a trusted home network.

⁠Configuration

Everything else — sources, models, launch specs, metrics retention — is configured at runtime from the Settings page, not via environment variables.

⁠Ports and volumes

• Port 8080 — the web UI and the /proxy/<source>/v1/... API endpoint.
• Volume at /data — Data Protection keys (/data/dataprotection). Mount this as a named volume or bind mount so login sessions survive container restarts.

⁠Using the proxy

This app isn't just a monitor — every source gets a stable OpenAI-compatible proxy endpoint at:

http://<host>:8080/proxy/<slug-or-id>/v1

Point any OpenAI-compatible client (LibreChat, OpenWebUI, opencode, a raw SDK) at that URL instead of the backend directly. Requests are forwarded to whatever the source's backend actually has loaded; if the client asks for a different model than what's currently running (and hot-swap is available — see below), the proxy swaps the backend container first and then forwards the request, so the client never needs to know a swap happened. /v1/models on that same endpoint returns every model registered for the source, not just whatever's currently loaded — this is what lets a client like LibreChat (with fetch: true in its endpoint config) auto-discover every available model instead of you hand-listing them.

Everything needed to make a source's proxy URL work is set in Settings → Server, on that source's own edit form:

• Label — also the default proxy slug (see below) and what shows up throughout the UI.
• Proxy slug — the <slug-or-id> segment of the URL above. Auto-generated from the Label if left blank; set it explicitly if you want a stable URL that survives renaming the source later.
• Inference URL — the backend's own /v1 (vLLM) or base URL (Ollama) that the proxy actually forwards requests to.
• Metrics URL — the backend's /metrics endpoint (vLLM only), which is what powers the live monitoring/history features but isn't needed for the proxy itself.

Auto-swap-on-request, idle-unload, and startup preload additionally require the source to be a Docker-backed vLLM instance with a Physical Server configured (see below) — without that, the proxy still forwards requests, but can't swap or unload the backend container for you.

⁠Hot-swap requirements

To use model hot-swap, idle-unload, startup preload, or the auto-swap proxy on a given source, that source's vLLM instance must run in a Docker container, and that Docker host's Engine API must be reachable from the dashboard container. The Docker host, GPU vendor/VRAM, and shared Hugging Face cache directory are configured once per Physical Server (Settings → Server → "Physical Servers"), since they're properties of the machine, not any one vLLM instance — multiple vLLM instances on the same box share one Physical Server entry, so the same weights are never downloaded twice. Each vLLM instance (source) then just picks which Physical Server it runs on, plus its own container name/image/port. Sources you only want to monitor don't need any of this.

By default, Docker only listens on its local Unix socket — the Engine API isn't reachable over the network until you turn that on. On a systemd-based Linux host (Ubuntu, Debian, Fedora, etc.), enabling it means adding a systemd override rather than editing docker.service directly (that file gets regenerated by package updates):

# 1. See the current ExecStart line so you don't lose any existing flags
#    (commonly something like --containerd=/run/containerd/containerd.sock).
systemctl cat docker.service

# 2. Create a drop-in override. Adjust the ExecStart line below to include
#    whatever flags step 1 showed, plus -H tcp://0.0.0.0:2375.
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/override.conf <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock
EOF

# 3. Apply it.
sudo systemctl daemon-reload
sudo systemctl restart docker

# 4. Verify from another machine on your network.
curl http://<host>:2375/version

The blank ExecStart= line before the real one is required — systemd appends to ExecStart by default, so without it you'd end up running dockerd twice.

This exposes full, unauthenticated control of the Docker host to anyone who can reach that port — there's no login, and anyone who can hit 2375 can run arbitrary containers, mount the host filesystem, etc. Treat it like exposing SSH with no password. At minimum, firewall port 2375 to only the dashboard's own IP; for anything beyond a trusted home LAN, use TLS on port 2376 with client certificates instead (Docker's own docs cover generating those) and point the source's Docker host at tcp://<host>:2376 once configured. Provisioning TLS certs isn't handled by this app — it's a one-time manual setup step on the Docker host.