magiccodingman/vllm-radiance repository overview

⁠vllm-radiance

A vLLM inference-server image for the AMD Radeon AI PRO R9700 (gfx1201 / RDNA4). It combines a pinned vLLM v0.30.0 ROCm stack with libr4d⁠'s hand-written RDNA4 attention, gated-delta-net, vision, all-reduce, MXFP4, and DFlash kernels while retaining Radiance's tuned FP8 GEMM and speculative-decoding paths.

The source platform is vLLM 0.30.0. Source merges do not publish or deploy a replacement for production 1.0.16. Its native FP8 and Quark27B TP2/C1 resident evidence, actual Runner V2/kernel selection, ordinary vision and NVFP4 conversion limits are recorded in V030_UPGRADE.md⁠. Fresh v0.30 publication measurements and their qualification limits are below. NVFP4→MXFP4 is default-off load-time requantization, not native NVFP4 execution or model-level quality qualification. Unqualified NVFP4A16 conversion is explicitly rejected; quantized-input NVFP4 remains eligible for the opt-in W4A8 conversion.

This fork tracks and credits DeadCode's vllm-radiance⁠ and libr4d work, with additional compiler pins, upstream v0.30 DFlash2 and structured-output ownership, native gfx1201 MXFP4/W4A8 support, reproducible benchmarks, and deployment qualification. Published images are at magiccodingman/vllm-radiance⁠.

Two additional opt-in laboratories preserve the qualified default: immutable, checkpoint-bound FP8 attention/KV calibration sidecars for fidelity work, and an offline collect → tune → verified-serve PyTorch TunableOp workflow for residual BLAS GEMMs. Neither is enabled until its exact model/profile passes the normal correctness and benchmark gates. See FP8-KV calibration and persisted TunableOp⁠.

⁠Quick start

The portable Compose file contains no machine-local paths. Copy the environment template and point it at your model directory:

git clone https://gitlab.sayou.io/lance-wright/vllm-radiance.git
cd vllm-radiance
cp .env.example .env
# Edit MODELS, MODEL_PATH, and SERVED_MODEL_NAME in .env.
mkdir -p vllm-cache
docker compose up -d
docker compose logs -f

The reusable baseline is native FP8 weights, FP8 KV, TP2, 16K maximum context, 85% GPU allocation, an eight-request admission ceiling, and automatic prefix caching with hybrid-GDN state alignment. It listens on 0.0.0.0:8000, retains language and vision support, enables Qwen tool/reasoning parsers, loads the checkpoint-native chat template and generation defaults, and allows clients to override request-level sampling and reasoning effort.

MAX_NUM_SEQS is an admission ceiling—not a promise that every admitted request can simultaneously reach MAX_MODEL_LEN. Select both from the measured capacity tables below.

Common operations:

docker compose up -d
docker compose ps
docker compose logs -f vllm
curl -fsS http://localhost:8000/health
docker compose down

Host paths, GPU IDs, private image tags, and local overrides belong in the gitignored .env or an ignored docker-compose.dev.yml, never in the public Compose file. See .env.example and docker-compose.dev.example.yml in the source repository⁠.

⁠Target formats

⁠Native FP8

The default Compose profile expects a native-FP8 checkpoint:

WEIGHT_QUANTIZATION=fp8
GPU_UTIL=0.85
MAX_MODEL_LEN=16384
MAX_NUM_SEQS=8

Radiance keeps its preshuffled block-FP8 dispatcher, fused RMSNorm/FP8 quantization, split-K fixes, R4D attention/GDN, and custom TP2 all-reduce. Replacing the FP8 dispatcher with the generic upstream AITER linear path was 9–11% slower in matched controls.

⁠AMD Quark MXFP4 with native W4A8

For amd/Qwen3.8-27B-Quark-AWQ-MXFP4⁠, point MODEL_PATH at the checkpoint and use:

WEIGHT_QUANTIZATION=auto
RADIANCE_MXFP4=1
RADIANCE_MXFP4_W4A8=1
RADIANCE_MXFP4_W4A8_MIN_M=0
RADIANCE_MXFP4_DECODE_MAX_M=64
RADIANCE_MXFP4_TN4_MIN_M=2048
RADIANCE_MXFP4_WPERM=1
RADIANCE_MXFP4_DECODE_NT=1

auto lets vLLM consume the checkpoint's Quark metadata. On gfx1201, W4A8 retains packed OCP group-32 MXFP4 weights and dynamically quantizes activations to FP8 E4M3 so the kernels use RDNA4's native FP8 WMMA path. Keep RADIANCE_MXFP4_W4A8_MIN_M=0: the generic AITER W4A4 fallback is numerically incorrect for one of the qualified Qwen GDN projections. The decode-shaped kernel covers M<=64; larger batches use the prefill kernel.

The final two switches are the qualified RX5-safe decode subset. They store weights in the kernel's fragment order and use non-temporal decode loads. The more aggressive RX5 A-tiled, norm-quant, and FP8-stream paths remain available only as disabled experiments; do not infer that RX5 as a whole is qualified.

The checkpoint's embedded MTP tensors are BF16. Add RADIANCE_QUARK_BF16_MTP=1 only when selecting its MTP profile. Non-spec and DFlash do not need that override.

Implementation, numerical controls, provenance, and immutable runs are documented in MXFP4/W4A8 on dual R9700⁠.

The later ggz14 RX4 traced-quant and FP8 residual-stream kernels are included but remain off by default. On this dual-R9700 qualification they produced only a mixed +2.1% weighted single-stream signal, regressed ITL 1%-low by 21%, did not improve c1/c2/c8 or prefill, and failed strict greedy/tool-call gates. Do not enable RADIANCE_NORMQUANT_FUSION or RADIANCE_FP8_STREAM in production; see the RX4 continuation report⁠. The subsequent safe-kernel selection, FP8-KV calibration work, and complete RX5 negative results are recorded in the RX5 continuation report⁠.

⁠Serving modes

Choose exactly one mode. RADIANCE_SPECULATIVE_CONFIG contains either MTP or DFlash; the modes are not cumulative.

⁠Fast MTP

For a Qwen checkpoint with an in-checkpoint MTP head:

RADIANCE_SPECULATIVE_CONFIG='{"method":"mtp","num_speculative_tokens":8,"attention_backend":"R4D","disable_padded_drafter_batch":true}'
RADIANCE_FAST_DRAFT=1

K8 is a ceiling. Radiance's dynamic controller may select a shallower depth based on confidence and active batch size. Fast draft uses an INT2-g128 LM-head copy with exact top-64 reranking; target verification remains in place.

⁠DFlash2

For the native-FP8 ARA target and its selective-FP8 drafter:

MAX_MODEL_LEN=8192
VLLM_USE_V2_MODEL_RUNNER=1
RADIANCE_COMPILATION_CONFIG='{"cudagraph_mode":"PIECEWISE"}'
RADIANCE_FAST_DRAFT=1
RADIANCE_SPECULATIVE_CONFIG='{"method":"dflash","model":"/models/Qwen3.8-27B-heretic-ara-DFlash2-fp8-magiccodingman","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"attention_backend":"TRITON_ATTN","max_model_len":8192,"disable_padded_drafter_batch":true}'

For AMD's Quark MXFP4 target, use the target-matched tcclaviger/Qwen3.8-27B-DFlash2-FP8⁠ drafter. RADIANCE_FAST_DRAFT=1 runtime-quantizes eligible draft linears to W4 and uses the INT2 exact-rerank head. The target retains R4D attention while the drafter uses Triton attention. The current image also merges GDN input projections, uses libr4d's fused speculative GDN update, increases exact-rerank width to 64, and narrows DFlash verification per request only at c5 and above when observed acceptance says the full K7 target verification is wasteful. Every optimization is independently reversible through the controls documented in benchmarks/README.md.

Prefix caching and MAMBA_CACHE_MODE=align remain the deployment defaults. Disable them only for a cold, nonce-disjoint benchmark or a deliberate maximum-capacity experiment. Recreate the container after changing modes:

docker compose down
docker compose up -d

⁠Measured capacity on two 32 GiB R9700s

These values include FP8 KV, TP2, no CPU/KV offload, and deliberate VRAM headroom. They are model- and profile-specific; larger models and different drafters must be requalified.

⁠Native FP8 target plus selective-FP8 DFlash drafter

Measured at 85% GPU allocation with prefix caching disabled for the capacity laboratory:

Every submission completed; minimum observed physical headroom was 4.41 GiB per GPU.

⁠Quark MXFP4/W4A8 target plus matched DFlash drafter

Measured at 90% GPU allocation. The target payload is 18.44 GiB versus 28.75 GiB for the native-FP8 regression target, a 10.31 GiB (35.9%) reduction.

The recommended long-context deployment is 128K/C4, 90% allocation, prefix caching enabled, MAMBA_CACHE_MODE=align, and DFlash K7. The capacity qualification itself used K5 (the draft depth does not change the reserved model/KV capacity): it exposed 576,001 GPU KV tokens (4.39 full 128K requests), completed four simultaneous full-context requests without OOM or preemption, and retained 5.17 GiB minimum physical headroom per GPU. A repeated 32K prefix reduced TTFT from 9.04 seconds cold to 0.70–0.71 seconds warm. Prefer K5 only for a workload that remains dominated by steady c8 traffic.

Full methodology and run IDs are in the Compose capacity report⁠.

⁠Optional CPU KV offload

Set RADIANCE_KV_OFFLOADING_SIZE (GiB) only when the required long-context envelope exceeds the GPU KV tier. Radiance coordinates mmap host registration across TP workers and defaults RADIANCE_KV_OFFLOAD_PIN_POLICY=auto: every rank uses pinned DMA only when all ranks register successfully; otherwise the failed HIP error is drained and all ranks coherently use slower pageable DMA. Use required when silently losing pinned-transfer performance is unacceptable, or disabled as a diagnostic control.

RADIANCE_KV_OFFLOAD_REGISTER_CHUNK_GIB=0 is the shipped default and preserves one whole-region registration. Positive chunk sizes are experimental until qualified on the deployment host. The dual-R9700 32/36 GiB investigation and reproducible maintenance probe are documented in ROCm KV-offload registration hardening⁠.

⁠Measured performance

⁠Current vLLM 0.30 publication — 2026-09-22

BetterBench v0.2.2 / corpus v1, 10 measured passes/category, two warmups/category, greedy cold nonce prompts, TP2, FP8 KV, 8K/C8, 85% GPU allocation, 4,096 batch tokens, PIECEWISE, prefix off, safe WPERM/decode-NT profile. Same Quark target and matched tcclaviger DFlash2 revisions throughout; fixed depths, no tuning. Each mode completed 80/80 category measurements, 24/24 requests at each concurrency, and 12/12 prefill measurements.

Concurrency columns are aggregate TPS, not per-request TPS. All speculative modes matched only 1/8 strict non-spec fixed outputs; K5/K7 matched 6/8 each other. Non-spec and both DFlash lanes passed 30/30 sampled required-tool checks. MTP passed 29/30: one unfinished tool JSON entered a whitespace loop and hit its 1,024-token limit. Its numbers are retained as experimental measurements, not successful tool-serving qualification. No gate was relaxed. Non-spec is the safe recommendation; K7 is the fastest measured single-stream option only when these speculative limitations are acceptable. K5 performed better at c8.

Category medians for the recommended control and fastest experimental lane:

Standard cold-prefill throughput (prompt tokens / TTFT):

Actual median prompt lengths were 1,556 / 3,023.5 / 5,226, not exact 2K/4K/7K. See the publication report⁠ for every mode's category table, acceptance, per-request concurrency TPS, caveats and immutable artifacts. These are current platform results; older source/profile results below are historical context, not an isolated “v0.30 improved by X%” experiment.

⁠Historical v0.28 / prior Radiance results

BetterBench v0.2.2 used its v1 corpus, ten measured passes per category, greedy decoding, cold nonce-prefixed prompts, and c1/c2/c4/c8 on two R9700s. The historical safe RX5 MXFP4 kernel profile adds RADIANCE_MXFP4_WPERM=1 and RADIANCE_MXFP4_DECODE_NT=1 while keeping full RX5 (A_TILED, GDN_NORM_QUANT, NORMQUANT_FUSION, and FP8_STREAM) disabled. The measured serving lane used the matched DFlash K7 drafter, TP2, FP8 KV, and PIECEWISE graphs:

Single-stream category medians:

Cold prefill measured 4,031.8 / 4,444.7 / 4,268.6 TPS at the 2K/4K/7K target depths. Every concurrency arm completed 24/24 requests. These are the standard 8K/C8 laboratory results at 85% GPU allocation with prefix caching and CPU offload disabled; the 128K/C4 production profile above intentionally has a different capacity/latency contract. Exact category TTFT, ITL, prefill, run metadata, and immutable raw results are in the historical safe-subset BetterBench report⁠ and RX5 continuation report⁠.

DFlash remains experimental and opt-in because strict speculative/non-spec greedy equivalence has not passed, even though the stable-default lane passed its meaningful-output and sampled tool-call qualification. The full RX4/RX5 traced-quant, tiled-prefill, GDN norm-quant, and FP8 residual-stream profile is not represented by the table above and remains off. The full RX5+DFlash interaction passed only 93/100 tool calls; constraining it to one tool call improved that to 98/100 but did not qualify it.

For historical mode-to-mode context, the earlier Radiance 0.9.3/libr4d 0.5.0 matched publication measured:

This older table predates the current v0.30 publication above and uses a different source/profile. Do not treat it as the current K7 performance ceiling. Its per-category TPS, acceptance, TTFT/TPOT, prefill, telemetry, confidence intervals, negative results, and immutable run IDs are in the Radiance 0.9.3 qualification report⁠.

⁠What is included

• libr4d 0.5.0: RDNA4 attention, GDN prefill/decode/spec-state handling, vision flash attention, exact and rotated-six-bit TP2 all-reduce, BF16/DFlash GEMMs, and DFlash-specific kernels.
• Radiance FP8 paths: preshuffled block-FP8 GEMMs, split-K alignment fixes, fused RMSNorm/quantization, and guarded fallbacks.
• Native Quark MXFP4/W4A8: packed OCP group-32 weights with dynamic FP8 activation quantization and separate small-M decode and prefill kernels.
• Fast speculative drafting: dynamic MTP depth, verbatim n-gram tails, INT2 exact-rerank heads, and W4 DFlash draft linears.
• Hybrid-safe prefix caching: automatic prefix caching with GDN convolution/recurrent-state restoration through --mamba-cache-mode=align.
• Spec-safe structured output: upstream XGrammar termination and reasoning-boundary fixes prevent speculative draft batches from overrunning or desynchronizing the tool-call grammar; Qwen structural-tag normalization also preserves open nested objects used by generic deferred-tool wrappers.
• Topology qualification: a background startup sweep reports GPU enumeration, P2P access, NUMA distance, and peer-copy bandwidth.

Unsupported geometries fall back per operator. AITER, FLA, Triton, and RCCL controls remain available for matched experiments.

⁠Build

The published image is built entirely from pinned source commits:

docker build \
  -t vllm-radiance:$(cat VERSION) \
  --build-arg RADIANCE_VERSION=$(cat VERSION) \
  .

The multi-stage build compiles the stack for gfx1201, prunes unrelated ROCm device code, builds libr4d with the image's hipcc, and copies only the runtime into the release stage. A compiler and headers remain in the release image because AITER JIT-compiles kernels on first use. The pruned image measured 3.66 GiB compressed, down from 9.35 GiB before pruning. A full build takes hours; Dockerfile.patch provides a guarded overlay for ordinary Radiance/libr4d iteration without rebuilding PyTorch and the compiler stack.

Do not independently bump PyTorch, Triton, torchvision, or vLLM. The qualified versions are a compiler stack, and an earlier mismatched combination caused sustained TP hangs.

⁠Documentation

• Upgrade and reproducibility history⁠
• Stable vLLM v0.28 upgrade and qualification⁠
• Radiance 0.9.3 / libr4d 0.5.0 qualification⁠
• MXFP4/W4A8 implementation and validation⁠
• RX4 MXFP4 continuation and qualification⁠
• Compose capacity and prefix-cache qualification⁠
• DFlash2 optimization and correctness investigation⁠
• XGrammar speculative-decoding backport⁠
• Qwen open nested-object tool-call fix⁠
• BetterBench methodology and earlier mode comparison⁠
• Benchmark laboratory usage⁠
• Complete runtime knob reference⁠

The source repository is the canonical location for detailed qualification evidence. This landing page is intentionally concise so the same content can be published as the Docker Hub repository overview.

⁠Upstream and attribution

This fork exists on top of two unusually strong RDNA4 efforts:

• StillDeadcode/vllm-radiance⁠ and StillDeadcode/libr4d⁠ provide the core Radiance runtime and hand-written gfx1201 kernels.
• ggz14/radiance-vllm-mxfp4⁠, authored by Brian, is the source of the native Quark MXFP4/W4A8 work and the RX3/RX4 optimization series adapted here. Its original authorship is preserved in the Git history.

The continuation pins the exact audited ggz14 upstream commit in its qualification report. Changes are ported selectively because this fork carries a different vLLM/libr4d base and additional DFlash and correctness patches; attractive results from incompatible or failed experiments are not silently copied.