havenoammo/llama repository overview

⁠llama.cpp with Multi-Token Prediction (MTP) Support

Docker images for llama.cpp⁠ built from the latest main branch, with full Multi-Token Prediction (MTP) support including image input and recent bug fixes not yet available in official releases.

⁠What is MTP?

Multi-Token Prediction (MTP) is a speculative decoding technique that allows the model to predict multiple tokens per forward pass, resulting in significantly faster inference with no quality loss. To use it, you need both an MTP-compatible server build and a model that includes MTP draft layers.

⁠Available Tags

⁠Compatible Models

⁠Unsloth MTP Models (quantized MTP layers)

• Qwen3.6-27B-MTP-GGUF⁠
• Qwen3.6-35B-A3B-MTP-GGUF⁠

⁠Q8 MTP Models (full Q8 MTP layers for improved prediction accuracy)

• havenoammo/Qwen3.6-27B-MTP-UD-GGUF⁠
• havenoammo/Qwen3.6-35B-A3B-MTP-GGUF⁠

The Q8 variants keep MTP layers at full Q8 quantization, which can improve speculative decoding accuracy at the cost of slightly more VRAM usage.

⁠Quick Start

docker run --gpus all --rm \
    -p 8080:8080 \
    -v ./models:/models \
    havenoammo/llama:cuda13-server \
    -m /models/Qwen3.6-27B-MTP-UD-Q8_K_XL.gguf \
    --port 8080 \
    --host 0.0.0.0 \
    -n -1 \
    --parallel 1 \
    --ctx-size 262144 \
    --fit-target 844 \
    --mmap \
    -ngl -1 \
    --flash-attn on \
    --metrics \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --jinja \
    --chat-template-kwargs '{"preserve_thinking":true}' \
    --ubatch-size 512 \
    --batch-size 2048 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --spec-type mtp \
    --spec-draft-n-max 3

Adjust as you see fit. What matters most for MTP is --spec-type mtp and --spec-draft-n-max 3.

⁠Switching from Official llama.cpp Docker Images

If you are already using the official ghcr.io/ggml-org/llama.cpp images, switching is straightforward. Simply replace the image name with the appropriate tag above. The server flags and model paths remain the same.

⁠Feedback

If you run into issues with a specific backend, feel free to open a discussion. Reports on which hardware and configurations work are especially helpful.