btbtyler09/vllm-rocm-gfx908 repository overview

This is a prebuilt vLLM docker container for MI100 (gfx908) GPUs.

⁠4/19/2026 Update (v0.19.2rc1):

• vLLM v0.19.2rc1 with AITER⁠ support, synced with ~1,520 upstream vLLM commits since v0.16.
• ROCm 7.2.1, PyTorch 2.10 (ROCm release/2.10), Triton ba5c1517 — matches AMD's vLLM ROCm base pins.
• ~10 gfx908-specific performance optimizations baked in (adaptive Flash-Decoding split-K, MI100 Triton tuning, skinny GEMM, C++ paged-attention on gfx908, block_size=32 default, FULL_DECODE_ONLY graphs, custom all-reduce with graph-capture bypass).
• Recommend launching with --attention-backend TRITON_ATTN — on this stack vLLM auto-selects ROCM_ATTN which is up to 2× slower at long context. See the "Getting the best performance" section below.
• Measured uplift vs v0.16.1.dev on Qwen3.5/3.6-35B-A3B-GPTQ at TP=4: +25–63% output throughput depending on concurrency; Long Context (16K) roughly doubles; TTFT drops ~94% at c=1.
• Set VLLM_ROCM_USE_AITER=1 to enable AITER's Triton-based kernels (RoPE, attention helpers). All other AITER flags are auto-configured for gfx908 — CK ops, FP8/FP4, and Unified Attention are automatically disabled.
• HSA_OVERRIDE_GFX_VERSION is not required with native gfx908 support, but setting it to 9.0.8 is harmless if you already use it.
• GPTQ (4-bit and 8-bit) and AWQ quantizations are supported. 8-bit GPTQ is consistently slightly faster than 4-bit on Qwen MoE on MI100 — use it if you have the VRAM. Use --dtype half for GPTQ.
• Benchmarks and detailed per-model reports: mi100-llm-testing⁠

⁠3/11/2026 Update (v0.16.1.dev):

• vLLM v0.16.1 with AITER support for gfx908.
• ROCm 7.0, PyTorch 2.9.1, Triton 3.4.0.
• Introduced the VLLM_ROCM_USE_AITER=1 entry point and auto-configuration for gfx908.

Known issues:

• AITER Unified Attention is disabled on gfx908 — it corrupts model state after sustained requests. The default path (Triton or ROCm C++ attention) is stable.
• GPTQ models require --dtype half (float16). bfloat16 will cause errors.
• Default attention-backend auto-selection on v0.19 picks ROCM_ATTN, which is fine for short-prefill-heavy workloads but up to 2× slower at long context than TRITON_ATTN. Pass --attention-backend TRITON_ATTN for most workloads (see below).

⁠Tags

⁠Instructions

Run the container with GPU access. Adjust render devices and GPU count for your hardware (renderD128 = GPU 0, incrementing from there).

docker run -it \
  --network=host \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device=/dev/kfd \
  --device=/dev/dri/renderD128 \
  --device=/dev/dri/renderD129 \
  --device=/dev/dri/renderD130 \
  --device=/dev/dri/renderD131 \
  --env VLLM_ROCM_USE_AITER=1 \
  --env HF_HOME=/huggingface \
  -v /home/{user}/.cache/huggingface:/huggingface \
  btbtyler09/vllm-rocm-gfx908:v0.19.2rc1 \
  bash

Once the container is started you can run models using vllm serve:

vllm serve /models/Qwen3.6-35B-A3B-GPTQ-8bit \
  --tensor-parallel-size 4 \
  --attention-backend TRITON_ATTN \
  --gpu-memory-utilization 0.92 \
  --max-model-len 32768 \
  --dtype half \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --skip-mm-profiling \
  --limit-mm-per-prompt '{"image": 2}'

• --tensor-parallel-size should match your GPU count (1, 2, or 4).
• --max-model-len can be increased if you have memory headroom, or decreased for single-GPU setups.
• --gpu-memory-utilization controls how much VRAM vLLM reserves (0.75 is conservative, 0.92–0.95 for production).
• --attention-backend TRITON_ATTN is the recommended setting on this image; see below.

⁠Getting the best performance

The v0.19 image contains ~10 gfx908-specific performance ports. A few notes for getting them to fully kick in:

1. Attention backend: vLLM auto-selects ROCM_ATTN (C++ paged attention) on gfx908 because this image enables it there. It works, but on most workloads TRITON_ATTN is faster — the adaptive Flash-Decoding split-K, MI100 launch-grid tuning, and decode TILE_SIZE=32 ports all live on the Triton path. Pass --attention-backend TRITON_ATTN unless your workload is dominated by short-prefill (≤1K input) throughput, where ROCM_ATTN has a slight edge.
2. KV cache block size: defaults to 32 on gfx908 (was 16). Don't override with --block-size 16.
3. CUDA graphs: FULL_DECODE_ONLY + compilation.mode=NONE are forced on gfx908 automatically. Set VLLM_MI100_TORCH_COMPILE=1 to override if you know what you're doing.
4. Custom all-reduce: enabled on gfx908 for eager paths; auto-bypassed during CUDA-graph capture (graph-captured paths fall through to pynccl/RCCL to avoid a HIP IPC NaN bug on CDNA1).
5. 8-bit GPTQ preferred: on Qwen MoE at TP=4, 8-bit is consistently a few percent faster than 4-bit at small concurrency and up to ~13% faster at c=32.

Typical output throughput we measure on Qwen3.6-35B-A3B-GPTQ-8bit with --attention-backend TRITON_ATTN, 4x MI100 TP=4, block_size=32, FULL_DECODE_ONLY graphs:

Long context (16K in / 1K out, c=4): 100 tok/s.