Prebuild for RTX 30xx and newer. Turboquant prebuild from fork of llama-cpp maintained by TheTom.
485
Run high-performance GGUF inference with CUDA 13.2 using llama.cpp and the Docker image sandichhuu/llama-cpp:cu132.
This setup is optimized for:
mmprojdraft-mtp)llama-server262144)Before starting, make sure you have:
Verify GPU access:
docker run --rm --gpus all nvidia/cuda:13.2.0-base-ubuntu24.04 nvidia-smi
services:
llama-cpp:
image: sandichhuu/llama-cpp:cu132
container_name: llama-cpp
ports:
- 8080:8080
privileged: true
ipc: host
volumes:
- ./models:/app/models
command: >
-m /app/models/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/Qwen3.6-35B-A3B-UD-IQ4_NL.gguf
--port 8080
--host 0.0.0.0
-t 12
-c 262144
--parallel 1
-fa on
--cache-type-k turbo3
--cache-type-v turbo2
-ngl 999
--kv-unified
--cpu-moe
--no-context-shift
-b 4096
-ub 4096
--no-mmap
--direct-io
--fit off
--jinja
--no-cache-prompt
--cache-ram 0
--spec-type draft-mtp
--spec-draft-n-max 2
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities:
- gpu
restart: unless-stopped
.
├── docker-compose.yml
└── models
└── unsloth
└── Qwen3.6-35B-A3B-MTP-GGUF
docker compose up -d
Check logs:
docker logs -f llama-cpp
The server exposes an OpenAI-compatible API on:
http://localhost:8080
Example request:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen",
"messages": [
{
"role": "user",
"content": "Hello"
}
]
}'
| Flag | Description |
|---|---|
-fa on | Enables Flash Attention |
-c 262144 | Sets context length to 262k |
--cache-type-k q4_0 | Quantized KV cache for lower VRAM usage |
--cache-type-v q4_0 | Quantized V cache |
--no-mmap | Fully loads model into memory |
--direct-io | Reduces page cache overhead |
--parallel 1 | Single inference stream |
--spec-type draft-mtp | Enables speculative decoding |
--spec-draft-n-max 2 | Number of speculative draft tokens |
--mmproj | Loads multimodal projection model |
--jinja | Enables chat template rendering |
This configuration is tuned for large-scale inference workloads:
Recommended GPUs:
You can download GGUF models from:
Docker Hub:
https://hub.docker.com/r/sandichhuu/llama-cpp
Pull manually:
docker pull sandichhuu/llama-cpp:cu132
Content type
Image
Digest
sha256:1eb857c04…
Size
1.8 GB
Last updated
4 months ago
docker pull sandichhuu/llama-cpp:turboquant