stilldeadcode/vllm-radiance repository overview

⁠vllm-radiance

vLLM inference server for the AMD Radeon AI PRO R9700 (gfx1201 / RDNA4). Bundles a working ROCm + PyTorch + Triton + AITER + vLLM stack with the RDNA4 patches and custom kernels needed to run vLLM on this card, so you don't have to build the stack yourself.

⁠Tested so far

Untested: any other model, non-FP8 weights, single GPU, more than two GPUs, non-R9700 hardware. The defaults below are a starting point for these setups, not a general recommendation.

Qwen3.8-27B-FP8 / Qwen3.6-27B-FP8 are what everything here is tuned around: 64 layers, 48 of them linear attention (GDN) and 16 full attention, hidden 5120, head_dim 256 with 6 query heads per KV head. That is exactly the geometry the hand-written kernels are compiled for, so the whole R4D library engages on them. The MTP head is inside the checkpoint, so speculative decoding needs no separate drafter. They are vision-language checkpoints -- pass --language-model-only to skip the vision tower when serving text.

Qwen3.6-35B-A3B-FP8. RDNA4-tuned fused-MoE configs and the skinny MoE-gate GEMM activate automatically. One requirement: with --mamba-cache-mode=align its attention block size is 2240 and align asserts block_size <= max_num_batched_tokens, so pass --max-num-batched-tokens >= 2240.

Gemma-4-31B-it-FP8 (e.g. RedHatAI/gemma-4-31B-it-FP8-block). Quantization is auto-detected; tuned block-FP8 GEMM configs load for its shapes. Text and vision both work. Not a GDN hybrid, so drop --mamba-cache-mode, and it uses its own chat template and parsers. Long-context prefill is tuned for its head-512 global layers (up to -38% TTFT at 64K, -46% at 120K; inert on other head sizes). It carries a lot of KV (60 layers), so at a given --gpu-memory-utilization it wants a smaller --max-model-len than the Qwen models. For a large decode speedup pair it with Google's drafter google/gemma-4-31B-it-assistant, whose head-512 layer needs "attention_backend":"ROCM_AITER_UNIFIED_ATTN" in the speculative config.

⁠Why this exists

vLLM's ROCm builds target datacenter cards (MI300 / CDNA). RDNA4 workstation cards like the R9700 (gfx1201) don't work out of the box: AITER isn't enabled for the arch, GPU enumeration fails, several kernels need patching, and the vendor attention and GEMM paths aren't tuned for RDNA4. This image pins a working combination, builds AITER from source for gfx1201, applies the fixes, and adds tuned kernels.

⁠Stack

Everything below is compiled from source for gfx1201 in the image build (0.5.0 onward); nothing is pulled from a prebuilt wheel index.

The PyTorch / Triton / torchvision versions are the ones upstream builds vLLM against on ROCm, not a newer combination chosen for this image. Read the ROCm numbers, not pyproject.toml: 0.27.1's build-system asks for torch == 2.13.0, which is the CUDA build, while upstream's own ROCm image builds torch release/2.11 with torchvision 0.24.1 and pins no torch in requirements/rocm.txt. That distinction is deliberate: see the tensor-parallel hang note below.

transformers is pinned because vLLM does not pin it -- requirements/common.txt asks only for transformers >= 5.5.3, so an unpinned rebuild picks up whatever is newest and the stack moves underneath the build. transformers 5.15.0 made Gemma-4's head_dim a per-layer attribute and turned the global read into an exception that no released vLLM handles, so a Gemma-4 checkpoint fails during argument parsing, before a model or an attention backend exists. 5.14.1 is the last release before that change and loads every architecture this image serves. If you build your own image, keep the pin.

There is no flash-attention package: the vendor flash kernels have no gfx1201 device code. Attention runs on the AITER unified path, and the vision tower on the image's own Triton flash kernel.

⁠What it patches (to make vLLM run on gfx1201)

• GPU enumeration (amdsmi init order). Without it the platform is undetected and device count reads 0.
• AITER enablement for gfx12x (upstream gates it to MI3xx).
• Triton driver activation for the GPU-less model-inspection subprocess.
• Native sampler fallback (AITER's top-k/top-p kernel doesn't build on RDNA4).
• Tool-parser streaming vs non-streaming consistency.
• from_json Jinja filter for tool-calling chat templates.
• MTP drafter unpadding, so --speculative-config's disable_padded_drafter_batch:true works (the single-stream MTP speed path).
• MTP drafter multimodal mask alignment, so speculative decoding works with image inputs (otherwise the vision-placeholder mask outlives the compacted draft batch and the engine crashes).
• torch.compile telemetry JSON encoding, which otherwise raises TypeError: Object of type function is not JSON serializable at startup on this torch version (harmless but alarming: the serve came up anyway).

⁠Custom kernels and tuning (on by default, env-gated)

The hand-written kernels are a separate library, libr4d⁠, written for gfx1201 rather than for any one model. Each entry point is named for the geometry it is compiled for and refuses anything else, so import r4d; r4d.kernels() inside the image lists exactly what it covers. The build clones a pinned tag and compiles it with its own hipcc.

R4D attention (--attention-backend R4D, opt-in). Purpose-built attention kernels for this GPU instead of the tuned AITER path. The core is a transposed score matrix, S^T = K.Q^T: a wave32 matrix-core fragment splits a 16x16 tile column-wise, so with the score matrix transposed each lane owns exactly one query row and the softmax never leaves the lane. Measured in the serve against ROCM_AITER_UNIFIED_ATTN, same image and flags: +14.6% prefill throughput at 64K context (65.6K-token prompt, 21.25 s -> 18.54 s to first token), +4.1% at 16K, decode unchanged within noise. The gain scales with context because attention's share of prefill does -- 34% of prefill GPU time at 64K, 11.8% at 16K -- so this is a TTFT feature, not a tokens/s one. It is also more accurate than what it replaces: 1.69e-03 relative error against an fp32 oracle, against 2.28e-03.

Shape support is narrow on purpose: head_dim 256, paged block 16, 6 query heads per KV head, causal decoder attention, bf16 query, bf16 or fp8_e4m3 KV. Anything else is refused at startup with the reason and the backends that would work instead. With speculative decoding, give the drafter the same backend: "attention_backend":"R4D" in the speculative config.

All of these are baked ON in the image. Set RADIANCE_DYNAMIC_DRAFT=0 to turn draft control off (RADIANCE_DRAFT_SCHEDULE and RADIANCE_DRAFT_TAU are values, not toggles). RADIANCE_DYNAMIC_DRAFT only does anything when speculative decoding is enabled; it is lossless (it changes only how many tokens are drafted and whether they come from MTP or a verbatim copy of earlier text, never what the model verifies).

NUMA pinning (RADIANCE_NUMA_BIND / --numa-bind, off by default). On a multi-NUMA-node host, pin the server and its TP workers to the node(s) local to the GPUs. auto detects from the visible GPUs; SPEC may also be explicit nodes, bind=, interleave, preferred= or none. No-op on single-node hosts; needs --cap-add SYS_NICE under Docker's default seccomp.

⁠Requirements

• AMD Radeon AI PRO R9700 (gfx1201). Compiled for gfx1201 only, won't run on other GPUs. Two GPUs (TP=2) is the only configuration tested so far.
• Linux host with the amdgpu kernel driver and /dev/kfd + /dev/dri. ROCm userspace is inside the image.
• Docker with device passthrough.

⁠Run

On start the image prints a banner and a short preamble (GPU count, gfx1201 check, P2P, enabled optimizations, component versions), then hands off to vllm serve. First argument is the model path, the rest are vllm serve flags. The RADIANCE_* vars below are already baked ON in the image; they are spelled out here only so they are visible and easy to flip off.

docker run --rm -it \
  --device /dev/kfd --device /dev/dri \
  --group-add "$(getent group render | cut -d: -f3)" \
  --group-add "$(getent group video  | cut -d: -f3)" \
  --shm-size 4g --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  -v /path/to/models:/models:ro \
  -v "$PWD/vllm-cache:/cache" \
  -p 8000:8000 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e VLLM_ROCM_USE_AITER=1 -e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
  -e VLLM_ROCM_USE_AITER_MHA=0 -e VLLM_ROCM_USE_AITER_MLA=0 -e VLLM_ROCM_USE_AITER_MOE=0 \
  -e VLLM_ROCM_USE_AITER_LINEAR=0 -e VLLM_ROCM_USE_AITER_FP8BMM=0 \
  -e VLLM_ROCM_USE_AITER_FP4BMM=0 -e VLLM_ROCM_USE_AITER_RMSNORM=0 \
  -e NCCL_PROTO=Simple \
  -e RADIANCE_PRESHUFFLE=1 \
  -e RADIANCE_USE_R4D_AR=1 \
  -e RADIANCE_USE_R4D_AR_QUANT=1 \
  -e RADIANCE_FUSE_RMS_QUANT=1 \
  -e RADIANCE_DYNAMIC_DRAFT=1 \
  -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
  -e TRITON_CACHE_DIR=/cache/triton -e AITER_ROOT_DIR=/cache/aiter \
  -e TRITON_CACHE_AUTOTUNING=1 \
  stilldeadcode/vllm-radiance:0.9.3 \
    /models/YourOrg/Your-Model-FP8 \
    --served-model-name my-model \
    --quantization fp8 --kv-cache-dtype fp8 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.92 \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --enable-prefix-caching --mamba-cache-mode align \
    --speculative-config '{"method":"mtp","num_speculative_tokens":8,"attention_backend":"ROCM_AITER_UNIFIED_ATTN","disable_padded_drafter_batch":true}' \
    --no-async-scheduling \
    --host 0.0.0.0 --port 8000

Test:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"my-model","messages":[{"role":"user","content":"Hello!"}]}'

⁠compose

A ready-to-edit docker-compose.yml with the same flags, every device / group / volume already wired, and each knob commented lives in the source repo: https://codeberg.org/StillDeadcode/vllm-radiance⁠.

⁠First run is slower

With an empty cache the first start spends a few extra minutes compiling Triton / inductor kernels before the engine comes up; it looks idle but it is compiling. (Older builds spent 15 to 20 minutes here, dominated by the gated-delta-net fp32 autotune; that path is gone on this stack.) Mount a persistent cache so restarts stay fast:

  -v /path/to/vllm-cache:/cache \
  -e VLLM_CACHE_ROOT=/cache/vllm \
  -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
  -e TRITON_CACHE_DIR=/cache/triton \
  -e AITER_ROOT_DIR=/cache/aiter \
  -e TRITON_CACHE_AUTOTUNING=1

⁠Flags

Speculative decoding (MTP). Two forms depending on where the MTP head lives:

# Qwen3.8-27B / Qwen3.6-27B / 35B: the MTP head is in the target checkpoint, so no separate drafter
--speculative-config '{"method":"mtp","num_speculative_tokens":8,"attention_backend":"ROCM_AITER_UNIFIED_ATTN","disable_padded_drafter_batch":true}'

# ...and on the 27B hybrids, give the drafter the same R4D backend if the target uses it
--attention-backend R4D --speculative-config '{"method":"mtp","num_speculative_tokens":8,"attention_backend":"R4D","disable_padded_drafter_batch":true}'

# Gemma-4-31B: the drafter is a separate model, so add "model" (and --trust-remote-code --no-async-scheduling)
--speculative-config '{"method":"mtp","model":"/models/google/gemma-4-31B-it-assistant","num_speculative_tokens":8,"attention_backend":"ROCM_AITER_UNIFIED_ATTN","disable_padded_drafter_batch":true}'

What num_speculative_tokens means here. Under mtp with RADIANCE_DYNAMIC_DRAFT=1 (baked on) it is a ceiling, not a fixed cost: the controller drafts up to that many, stops early on low-acceptance content, and may take a verbatim n-gram continuation. So 8 is a good default -- it gives the drafter room on code and JSON without adding fixed overhead on prose. With RADIANCE_DYNAMIC_DRAFT=0, or under dflash, it is a fixed width again and a smaller value is typical.

DFlash drafters. dflash is a different speculative shape from mtp: instead of running one draft forward per speculative position, it emits all of them in a single pass, so the draft is one CUDA-graph replay whose depth is fixed at capture. That has three consequences worth knowing.

--speculative-config '{"method":"dflash","model":"/models/<drafter>","num_speculative_tokens":7,"attention_backend":"TRITON_ATTN"}'

• RADIANCE_DYNAMIC_DRAFT does nothing here (see the table above) -- num_speculative_tokens is a fixed width again, so it is worth tuning: on Qwen3.8-27B the per-position acceptance rate falls to ~0.10 by the seventh, and each extra position widens both the draft pass and the target's verify.
• The drafter's attention backend must be one that supports full CUDA graphs, or vLLM logs that it is "running the draft eagerly" and you lose the graph. TRITON_ATTN does; check the startup log for Capturing dflash CUDA graphs (FULL).
• Prefix caching with --mamba-cache-mode align has only been verified against mtp on hybrid models; it is a different speculative shape and has not been re-verified here.

disable_padded_drafter_batch:true is the key single-stream lever (~+50% on the 27B hybrids): it drops the drafter's batch padding, and the image bakes the vLLM unpad patch this relies on. Leave it on. Note it is incompatible with async scheduling: pass --no-async-scheduling to disable it explicitly (otherwise vLLM auto-enables async scheduling and then disables it with a runtime warning; --async-scheduling would hard-error).

Prefix caching (shared system prompts, RAG, agentic context):

--enable-prefix-caching --mamba-cache-mode align

Automatic prefix caching reuses a shared prompt prefix across requests so only the new suffix is prefilled -- a large TTFT drop when requests share a system prompt or document. On a GDN hybrid you must pass both flags: hybrid models default their prefix-caching support flag off, so vLLM silently disables it without --enable-prefix-caching, and align is what makes the linear-attention layers cacheable by snapshotting their conv + recurrent state at block boundaries. That restore is verified bit-identical to a full recompute, so outputs are unchanged and the win is purely latency (~3.6x faster TTFT on shared prefixes). Trade-offs: align raises the attention block size to 1664 tokens and adds one state block per linear-attention layer, so prefix hits land on 1664-token boundaries and max concurrency at full context drops slightly. Do not use --mamba-cache-mode all (unsupported, raises at startup) or set VLLM_SSM_CONV_STATE_LAYOUT=DS (asserts under MTP + align).

Tool-calling and reasoning:

--enable-auto-tool-choice --tool-call-parser <parser> --reasoning-parser <parser>

Pass a template with --chat-template file.jinja if the model needs one. The image ships the from_json filter those templates often rely on.