nullata/llamaman repository overview

⁠ llamaMan

What it is. A browser UI + API front-end for running multiple llama.cpp⁠ server instances from one Docker container. Ollama-compatible proxy so it drops in for Open WebUI⁠. What it is. A browser UI + API front-end for running multiple llama.cpp⁠ server instances from one Docker container. Ollama-compatible proxy so it drops in for Open WebUI⁠.

What's different. llamaMan has no llama.cpp code and no GPU dependency - it spawns ghcr.io/ggml-org/llama.cpp:server-* containers as siblings over the Docker socket. Every launch knob (spec-decoding, MoE offload, KV quant, flash-attn, load-mode, mmproj, PDF input) is a first-class UI field, and multi-node deployments can share a queue across heterogeneous hardware. Update llama.cpp without touching llamaMan: pull a newer server-* image from Settings → Docker Images.

Full docs and source: github.com/nullata/llamaman⁠

⁠Features

• Universal GPU image - one image, auto-detects NVIDIA / AMD (ROCm) / Intel Arc / CPU. Vulkan is opt-in via GPU_TYPE=vulkan for AMD/Intel hosts falling back off ROCm/SYCL.
• Model library + downloader - scans /models for GGUF, pulls from HuggingFace with speed limits, resume, auto-retry, and republish detection with atomic re-pull
• One-click launch + presets - per-model launch settings with live updates for the fields that don't need a relaunch (idle-timeout, gates, sampling overrides)
• Instance management - stop / restart / logs / stats; per-GPU VRAM, container CPU% + RAM, and per-instance throughput / TTFT / latency rolled up from the request log
• Ollama + OpenAI proxy on :42069 - Open WebUI drops in, auto-starts models on demand, LRU-evicts once LLAMAMAN_MAX_MODELS is hit
• Flash Attention + KV cache quant + reasoning format + load mode - --flash-attn, --cache-type-k/v, --reasoning-format, --load-mode (mmap/mlock/dio) all exposed; UI enforces the quantized-V-requires-FA-On constraint
• Anti-Loop - DRY sampler (soft, sampling-time) + proxy-side output loop detection that watches the streamed text and hard-kills the turn when a large chunk repeats often enough. Both off by default, tuned per preset
• Speculative decoding - all five draft-model families (draft-simple/-mtp/-dflash/-dspark/-eagle3) with advanced knobs (n-min, p-split, p-min)
• MoE offload - --cpu-moe / --n-cpu-moe exposed as a sentinel int; shrinks big-MoE VRAM ~4-10x by pinning routed experts to CPU while attention / embeddings stay on GPU
• Image & PDF input - --mmproj for vision models, PDFs rasterized page-by-page (or inlined as text via the born-digital shortcut). Works on OpenAI image_url/file and Ollama images[]
• Auth - user accounts, API keys (bearer tokens), toggle for whether model endpoints require auth
• Persistent state, JSON or MariaDB - MariaDB unlocks multi-node clustering (shared dashboard, cross-node launch/download, shared-queue load balancing) and an optional local write-through mirror that keeps the node serving through a DB outage
• Request recording + logging dashboard - opt-in per-request or per-conversation recording with retention, plus a dedicated logging page
• 🆕 Knowledge base + MCP server - opt-in (MariaDB 11.8+): store documents as topics, search them by meaning with your own embedding model, and serve them to MCP clients at :42069/mcp/knowledge, with an optional private knowledge base per API key. See the Knowledge Base & MCP docs on GitHub⁠

⁠Tags

• latest, <version> - universal image, auto-detects GPU vendor

⁠Quick Start

Pull the llama.cpp server image for your GPU, then run llamaMan. HOST_MODELS_DIR / HOST_LOGS_DIR MUST be the absolute paths on the Docker host - llamaMan passes them to the daemon when spawning sibling llama-server containers.

docker pull <LLAMA_IMAGE>              # from the table above
docker network create llamaman-net

docker run -d \
  --name llamaman \
  --network llamaman-net \
  -p 5000:5000 -p 42069:42069 -p 8000-8020:9000-9020 \
  -v /path/to/models:/models \
  -v /path/to/data:/data \
  -v /path/to/logs:/tmp/llama-logs \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /sys/class/drm:/sys/class/drm:ro \
  -e LLAMA_IMAGE=<LLAMA_IMAGE> \
  -e HOST_MODELS_DIR=/path/to/models \
  -e HOST_LOGS_DIR=/path/to/logs \
  -e LLAMAMAN_NODE_NAME=srv1 \
  --restart unless-stopped \
  nullata/llamaman:latest

LLAMAMAN_NODE_NAME is required - the container refuses to start without it. Pick it once and keep it (changing later orphans stored state). See the full env-var list below.

For native NVIDIA VRAM monitoring (via nvidia-ml-py), also add:

  --gpus '"driver=nvidia,capabilities=utility"' \

⁠Docker Compose

services:
  llamaman:
    image: nullata/llamaman:latest
    ports:
      - "5000:5000"
      - "42069:42069"
      - "8000-8020:9000-9020"
    volumes:
      - /path/to/models:/models
      - /path/to/data:/data
      - /path/to/logs:/tmp/llama-logs
      - /var/run/docker.sock:/var/run/docker.sock
      - /sys/class/drm:/sys/class/drm:ro
    environment:
      - LLAMA_IMAGE=ghcr.io/ggml-org/llama.cpp:server-cuda
      - LLAMAMAN_NODE_NAME=srv1
      - HOST_MODELS_DIR=/path/to/models
      - HOST_LOGS_DIR=/path/to/logs
    # NVIDIA native GPU monitoring - uncomment on NVIDIA hosts.
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           capabilities: [utility]
    networks: [llamaman-net]
    restart: unless-stopped

networks:
  llamaman-net:
    driver: bridge
    name: llamaman-net

⁠First Launch

1. Open http://localhost:5000⁠
2. Create an admin account on /setup
3. Drop GGUF files into /models (or download via the UI)

⁠Ports & Volumes

⁠Environment Variables

Only the ones you'll typically touch. See the full reference on GitHub⁠ for every knob.

⁠OpenWebUI Integration

Point OpenWebUI at the Ollama proxy (with an API key when require_auth is on, which it is by default):

open-webui:
  environment:
    - OLLAMA_BASE_URL=http://llamaman:42069
    - OPENAI_API_BASE_URLS=http://llamaman:42069/v1
    - OPENAI_API_KEYS=llm-your-api-key-here

Models are listed by GGUF filename stem; set a per-model Display Name to have OpenWebUI show/accept a friendly name instead. In a cluster, live shared-queue group aliases are advertised as selectable models too, so a client can send the alias and have it routed to the least-loaded node. Group context length is reported as the min across live members cluster-wide, and any node answers /api/show / /v1/models for a group even when it has no local member of it - so OpenWebUI sees the truthful runtime ctx instead of falling back to its built-in default.

⁠Persistent State

JSON (default): zero-config, stored in /data. Fine for single-node.

MariaDB / MySQL: set DATABASE_URL. Tables auto-created on first connect.

CREATE DATABASE llamaman CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'llamaman'@'%' IDENTIFIED BY 'yourpassword';
GRANT ALL PRIVILEGES ON llamaman.* TO 'llamaman'@'%';

MariaDB unlocks two optional features:

• Local DB mirror - write-through copy in DATA_DIR/db_mirror/ keeps the node serving through a DB outage (inference, launches, presets, API keys, settings; only first-user /setup is blocked). Offline writes are journalled and replayed on reconnect. Off by default; needs a persistent DATA_DIR per node. Hugging Face tokens land on local disk in plaintext if mirroring is on.
• Multi-node clustering - multiple llamaMan deployments share the DB and act as one logical cluster: aggregated dashboard, cross-node launch/pull/download, shared-queue load balancing. Requires CLUSTER_ENABLED=true, the same CLUSTER_SECRET on every node, and a per-node CLUSTER_ADVERTISE_URL for cross-node actions (a node without one is view-only).

⁠Per-Instance Proxy

When Idle Timeout, Max Concurrent, or Proxy Sampling Overrides are on for an instance, llamaMan puts a proxy in front of that instance's port. It handles auth, concurrency gating, wake-on-request, and model-name validation - requests with a "model" field are checked against the loaded model's filename stem (prefix match accepted; mismatch → 404, no wake for sleeping instances).

⁠Model Eviction

LLAMAMAN_MAX_MODELS caps concurrent chat models via the proxy. Embedding-flagged instances are excluded and never evicted; sleeping instances still count.

Three toggles under Settings → App Settings relax these defaults ("Enforce cap for admin UI launches", "Allow Ollama API to evict admin-launched", "Allow OpenAI API to evict admin-launched").

⁠Requirements

• Docker with access to /var/run/docker.sock
• One of: NVIDIA Container Toolkit⁠, a ROCm-compatible setup⁠, Intel Arc with /dev/dri access, a Vulkan-capable GPU with /dev/dri access (opt-in via GPU_TYPE=vulkan), or CPU-only

⁠Links

• Source: github.com/nullata/llamaman⁠
• Report Issues: github.com/nullata/llamaman/issues⁠
• Support the project: ko-fi.com/nickscripts⁠
• Website: nickscripts.com⁠

⁠License

Elastic License 2.0⁠