hellohal2064/llama-benchy repository overview

⁠llama-benchy

LLM inference benchmark tool with a built-in web dashboard. Test prompt processing (PP), token generation (TG), time-to-first-token (TTFT), and prefix caching performance against any OpenAI-compatible API endpoint.

⁠Quick Start

⁠Docker Run

docker run -d \
  --name llama-benchy-web \
  -p 5055:5000 \
  -v llama-benchy-data:/app/data \
  hellohal2064/llama-benchy:latest

Open http://localhost:5055⁠ in your browser.

⁠Docker Compose

version: "3.8"

services:
  llama-benchy:
    image: hellohal2064/llama-benchy:latest
    container_name: llama-benchy-web
    ports:
      - "5055:5000"
    volumes:
      - llama-benchy-data:/app/data
    restart: unless-stopped

volumes:
  llama-benchy-data:

docker compose up -d

⁠Usage

1. Open http://localhost:5055⁠ in your browser
2. Enter your OpenAI-compatible endpoint URL (e.g. http://192.168.4.208:8000)
3. Click Detect — the tool auto-detects the backend (vLLM, Ollama, llama.cpp, etc.), model, and tokenizer
4. Select a test group and click Start
5. Watch live progress via Server-Sent Events
6. View results, export as JSON, CSV, or Markdown

⁠Test Groups

⁠Features

• Auto-detection of backend type, model, and tokenizer
• Live progress with Server-Sent Events streaming
• Test history persisted to a named Docker volume — survives container restarts and rebuilds
• Delete individual runs from the web UI
• Export results as JSON (for Spark Arena), CSV, or Markdown
• Session persistence — endpoint config remembered between page loads
• Prefix caching analysis — measures warm vs cold TTFT across context depths

⁠Data Persistence

Test results are stored in the /app/data volume:

• *.json — Full benchmark result files
• runs/*.meta.json — Run metadata (config, status, timestamps)
• runs/*.log — Execution logs

Data persists across docker compose down / docker compose up. To wipe all data:

docker compose down -v

⁠CLI Mode

The image also supports direct CLI usage:

docker run --rm hellohal2064/llama-benchy:latest \
  llama-benchy --base-url http://your-server:8000 \
  --pp 2048 --tg 128 --concurrency 1

⁠Supported Backends

• vLLM — Full auto-detection including model path and tokenizer
• Ollama — Detects running model via /api/tags
• llama.cpp — Detects via /props endpoint
• Generic OpenAI-compatible — Any server with /v1/models

⁠Tags

• latest — Most recent build
• 0.3.1 — Current stable release