skyne98/tp906-engine repository overview

⁠tp906-engine

GPU inference engine for Qwen3.5 (Dense) hybrid models on AMD Instinct MI50 (gfx906).

Custom HIP kernels, flash attention, P2P tensor parallelism (TP=1..8), MTP speculative decoding. Reads GGUF model files.

⁠Quick Start

# Benchmark
docker run --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host -v /path/to/models:/models \
  skyne98/tp906-engine tp906-bench -m /models/model.gguf

# Chat
docker run -it --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host -v /path/to/models:/models \
  skyne98/tp906-engine tp906-chat /models/model.gguf

# OpenAI-compatible server
docker run --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host -v /path/to/models:/models -p 8080:8080 \
  skyne98/tp906-engine tp906-server /models/model.gguf --host 0.0.0.0

⁠Supported Models

Architecture: Qwen3.5 Dense (and MoE variants).

TP must evenly divide the model head count (32 for all dense Qwen3.5), so valid values are 1, 2, 4, 8, 16, 32.

⁠Supported Quantizations

⁠MTP (Multi-Token Prediction) Speculative Decoding

MTP provides ~10-15% faster decode by speculatively predicting one token ahead.

⁠Getting MTP Weights

Pre-extracted weights for all Qwen3.5 sizes: raspbfox/tp906-mtp⁠

# Download for your model size — place next to the GGUF file
cd /path/to/models/
wget https://huggingface.co/raspbfox/tp906-mtp/resolve/main/Qwen3.5-9B/mtp_weights.bin

Available: 0.8B⁠ (39 MB) | 2B⁠ (116 MB) | 4B⁠ (230 MB) | 9B⁠ (465 MB) | 27B⁠ (811 MB)

⁠Using MTP

Auto-detect: place mtp_weights.bin in the same directory as your GGUF file. tp906 finds it automatically.

Explicit: --mtp /path/to/mtp_weights.bin

Server INI: use the mtp key (see models.ini section below).

⁠Extracting MTP Weights Yourself

# Requires: pip install huggingface_hub numpy
python3 tools/extract_mtp.py --repo Qwen/Qwen3.5-9B -o mtp_weights.bin

Downloads only the needed safetensor shards, extracts the 15 MTP tensors, converts BF16 to F16.

⁠Multi-GPU (Tensor Parallelism)

docker run --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host -v /path/to/models:/models \
  skyne98/tp906-engine tp906-bench -m /models/model.gguf --tp 4

• P2P DMA with IPC shared memory -- --ipc=host is required
• Decode: P2P allreduce (low latency)
• Prefill: NCCL allreduce (high bandwidth)

⁠models.ini (Multi-Model Server)

Serve multiple models from one instance. Clients select via the model field in API requests. TP is automatically reconfigured when switching models.

; First entry loads at startup. Comments: ; or #

[Qwen3.5-9B]
path    = /models/Qwen3.5-9B-Q8_0.gguf
mtp     = /models/Qwen3.5-9B/mtp_weights.bin
tp      = 4
max_seq = 32768

[Qwen3.5-4B]
path    = /models/Qwen3.5-4B-Q8_0.gguf
tp      = 2

[Qwen3.5-0.8B]
path    = /models/Qwen3.5-0.8B-Q8_0.gguf
tp      = 1

docker run --device=/dev/kfd --device=/dev/dri --group-add video \
  --ipc=host -v /path/to/models:/models -p 8080:8080 \
  skyne98/tp906-engine tp906-server --models /models/models.ini --host 0.0.0.0

⁠CLI Reference

⁠tp906-bench

Benchmark tool with llama-bench compatible output format.

Comma-separated values for sweeps: -p 128,256,512 -n 64,128

⁠tp906-chat

Interactive chat CLI.

Interactive commands: /help /paste /file <path> /history /reset /quit

⁠tp906-server

OpenAI-compatible HTTP API server.

Endpoints:

⁠Requirements

• AMD Instinct MI50 (gfx906) GPU(s)
• ROCm kernel driver (amdgpu) on the host
• Docker flags: --device=/dev/kfd --device=/dev/dri --group-add video
• --ipc=host for multi-GPU tensor parallelism

⁠Recommended Host Tuning (Multi-GPU)

sudo sysctl -w kernel.numa_balancing=0   # prevent GPU DMA stalls
# Kernel parameter (GRUB): iommu=pt       # prevent IOMMU overhead

⁠Performance (MI50 x4, 1143 MHz comfort clocks)

⁠Version

0.0.1-pre-alpha-1 -- early testing release.