wdunn001/codec-vllm repository overview

⁠codec-vllm

vLLM inference server with Codec token-native binary transport plus the codec-supervisor control plane. One container, GPU-accelerated, OpenAI-compatible.

License: BUSL-1.1⁠ for the codec-supervisor wrapper. The bundled vLLM⁠ remains under Apache-2.0. Production use under BUSL-1.1 is permitted up to USD $5M annual gross revenue; above that, contact [email protected]⁠.

⁠Quick start

docker run -d --gpus all \
  -p 8080:8080 \
  -v codec-models:/models \
  -v codec-hf-cache:/root/.cache/huggingface \
  -e CODEC_INITIAL_MODEL=Qwen/Qwen2.5-0.5B-Instruct \
  --shm-size 8g --ipc host \
  wdunn001/codec-vllm:latest

# OpenAI-compatible
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"x","prompt":"Hello","max_tokens":20}'

# Codec wire format (msgpack frames of token IDs — ~200x smaller)
curl http://localhost:8080/v1/completions \
  -d '{"model":"x","prompt":"Hello","max_tokens":20,"stream":true,"stream_format":"msgpack"}'

⁠Run your own model

Three ways:

1. Override CODEC_INITIAL_MODEL — any HF repo id:

docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=meta-llama/Llama-3.1-8B-Instruct \
  -e HF_TOKEN=hf_xxx \
  wdunn001/codec-vllm:latest

2. Bind-mount a local checkpoint:

docker run --gpus all -p 8080:8080 \
  -e CODEC_INITIAL_MODEL=/models/my-finetune \
  -v /path/to/my-finetune:/models/my-finetune:ro \
  wdunn001/codec-vllm:latest

3. Hot-swap via the admin API — same surface as codec-sglang:

curl -X POST http://localhost:8080/admin/load \
  -H "Content-Type: application/json" \
  -d '{"name":"Qwen/Qwen2.5-7B-Instruct","allow_remote":true}'

Pass CODEC_BACKEND_ARGS to tune vLLM (--gpu-memory-utilization 0.9 --max-model-len 4096 --tensor-parallel-size 2 ...). Defaults are vLLM-shaped (not sglang-shaped).

⁠What's inside

• Base: vllm/vllm-openai:latest
• Codec patch: vllm-project/vllm#41765⁠ — token-native binary transport for completions and chat-completions streaming, applied as an editable overlay.
• Control plane: codec-supervisor⁠ — FastAPI admin on :8080 for model upload, HF pull, hot-swap, and an OpenAI-compatible reverse proxy.

⁠Source & docs

• Image source: https://github.com/wdunn001/codec-supervisor⁠ (see Dockerfile.vllm)
• Full docs: https://codecai.net/docs/⁠
• Codec protocol: https://codecai.net⁠
• Upstream PR: vllm-project/vllm#41765⁠