n8500x/llamacpp repository overview

⁠llamacpp

llama.cpp⁠ built from source at a pinned release tag, CPU-only, no model baked in — mount any GGUF at runtime. Exposes the same OpenAI-compatible API and env contract as n8500x/cpu-coder⁠ (which bakes a model in; this is the engine alone).

• Image: n8500x/llamacpp (slim — engine + CLI tools only)
• Pinned tag: b10884 (build arg LLAMA_TAG)
• Portability: compiled with GGML_CPU_ALL_VARIANTS + GGML_BACKEND_DL — every x86-64 CPU backend is built and the best one is picked at runtime, so one image runs optimally from old Xeons to AVX-512 hosts. GGML_NATIVE=OFF, so the build machine's CPU doesn't leak into the binary.
• Air-gap: nothing fetched at runtime (LLAMA_CURL=OFF).

⁠Serve a model

docker run -d --name llamacpp -p 8080:8080 \
  -v ./my-model.gguf:/models/my-model.gguf:ro \
  n8500x/llamacpp

Or docker compose up -d — the bundled docker-compose.yml⁠ carries the full annotated configuration (serving, prefill tuning, speculative decoding via the mtp carrier profile, memory sizing). The image is self-documenting — on any host, extract that config straight from it:

docker run --rm n8500x/llamacpp compose > docker-compose.yml
docker run --rm n8500x/llamacpp readme

The entrypoint serves the first /models/*.gguf it finds (or set MODEL_PATH).

Native LLAMA_ARG_* server env vars pass through as well.

Long prompts / agent skill files: the server-side pieces (prefix caching, CPU batch tuning, all-core prefill) are on by default, but caching only pays off if the client keeps the large stable text (system prompt, skill file) as the leading prefix of every request — anything before it that changes per request (timestamps, request IDs) forces a full re-prefill. Each parallel slot keeps its own cache, so with PARALLEL > 1 the server routes requests to the slot with the best prefix match; many distinct agents sharing few slots will still thrash. The container runs as non-root (UID 1001, group 0, g=u) and works under OpenShift's restricted-v2 arbitrary UIDs; mounted model files just need to be world- or group-readable.

⁠CLI tools

llama-cli, llama-bench, llama-quantize, and llama-gguf-split are baked in; naming one as the first argument bypasses the server:

docker run --rm -v ./m.gguf:/models/m.gguf n8500x/llamacpp llama-bench -m /models/m.gguf
docker run --rm -v .:/w n8500x/llamacpp llama-quantize /w/f16.gguf /w/q4_k_m.gguf Q4_K_M

⁠MTP draft carrier image

Dockerfile.mtp packages the Qwen3.6-35B-A3B MTP draft head (mtp-Qwen_Qwen3.6-35B-A3B-Q8_0.gguf, ~2 GB, sha256-verified at build) into n8500x/mtp-qwen3.6-35b-a3b, so the model travels through Docker Hub instead of Hugging Face — build/push where HF is reachable, docker pull where it isn't.

.\build-mtp.ps1 -Push     # downloads from HF in-build, verifies sha256, pushes

On the pulling side, either extract the file:

id=$(docker create n8500x/mtp-qwen3.6-35b-a3b true)
docker cp "$id":/models/mtp-Qwen_Qwen3.6-35B-A3B-Q8_0.gguf .
docker rm "$id"

or populate a shared volume (init-container style):

docker run --rm -v models:/target n8500x/mtp-qwen3.6-35b-a3b
docker run -d -p 8080:8080 -v models:/draft -e DRAFT_PATH=/draft/mtp-Qwen_Qwen3.6-35B-A3B-Q8_0.gguf \
  -v ./main.gguf:/models/main.gguf:ro n8500x/llamacpp

⁠Build

.\build.ps1          # clone at $LLAMA_TAG + compile (~15-30 min), tags date+latest
.\build.ps1 -Push    # also push both tags, records .push_tag

To bump llama.cpp: change LLAMA_TAG in the Dockerfile to a newer release tag⁠ and rebuild. The in-build git clone uses http.sslVerify=false (build-time only) because of host TLS interception; runtime makes no network calls.