wdunn001/codec-llamacpp repository overview

⁠codec-llamacpp

llama.cpp llama-server with Codec token-native binary transport plus the codec-supervisor control plane. One container, GPU-accelerated, OpenAI-compatible.

License: BUSL-1.1⁠ for the codec-supervisor wrapper. The bundled llama.cpp⁠ remains under MIT. Production use under BUSL-1.1 is permitted up to USD $5M annual gross revenue; above that, contact [email protected]⁠.

⁠Quick start

Default boot downloads Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M (~400 MB) and serves it.

docker run -d --gpus all \
  -p 8080:8080 \
  -v codec-models:/models \
  -v llamacpp-cache:/root/.cache/llama.cpp \
  --shm-size 8g \
  wdunn001/codec-llamacpp:latest

# OpenAI-compatible
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"x","prompt":"Hello","max_tokens":20}'

# Codec wire format (msgpack frames of token IDs)
curl http://localhost:8080/v1/completions \
  -d '{"model":"x","prompt":"Hello","max_tokens":20,"stream":true,"stream_format":"msgpack"}'

⁠Run your own model

Three ways:

1. HF GGUF id — Owner/Repo-GGUF:filename-glob:

docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL='Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M' \
  wdunn001/codec-llamacpp:latest

2. Bind-mount a local .gguf file:

docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=/models/my-model.gguf \
  -v /path/to/my-model.gguf:/models/my-model.gguf:ro \
  wdunn001/codec-llamacpp:latest

3. Hot-swap via the admin API:

curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name":"Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M","allow_remote":true}'

Pass CODEC_BACKEND_ARGS to tune llama-server (--ctx-size 8192 --gpu-layers 999 --threads 16 ...).

⁠What's inside

• Build: llama.cpp built from source (multi-stage nvidia/cuda:devel → runtime) against wdunn001/llama.cpp feat/codec-compression (stacked: PR #22757 + server-side ToolWatcher + streaming gzip).
• Codec patches: ggml-org/llama.cpp#22757⁠ (token-native binary transport) plus the follow-up commits.
• Control plane: codec-supervisor⁠ — FastAPI admin on :8080 for model upload, HF pull, hot-swap, and an OpenAI-compatible reverse proxy.

⁠Source & docs

• Image source: https://github.com/wdunn001/codec-supervisor⁠ (see Dockerfile.llamacpp)
• Full docs: https://codecai.net/docs/⁠
• Codec protocol: https://codecai.net⁠
• Upstream PR: ggml-org/llama.cpp#22757⁠