tcclaviger/vllm - ROCm/RDNA4 vLLM with native HIP kernels, baked-in quantizers + tuner
vLLM 0.30 tree on ROCm 10.0, torch 2.11, Python 3.14, built for gfx1201 (Radeon R9700). With no tool flag the image runs vllm serve "$@", so it drops into an existing vLLM deployment unchanged.
turbo is on by default for the Qwen4 architecture at TP2 and TP4, with and without expert offload, CLAV_Q4_TURBO=0 turns it off and runs the eager path
turbo on block fp8 checkpoints
turbo now serves the block fp8 checkpoints of Qwen3.8-Flash-Next, the expert GEMMs run inside the persistent launch and are bit identical to the eager fp8 path
the expert format is read from the checkpoint at startup, MXFP4 and block fp8 each have their own launch libraries and the serve picks the right one with no flag
the fp8 libraries read the experts through xmove, so an fp8 checkpoint whose experts do not fit the cards serves from host memory with the launch on
an fp8 checkpoint whose attention projections are stored in bf16 gets them quantized to fp8 block 128 at load, CLAV_PROJ_FP8_REPACK=0 keeps them bf16 and the launch off
the expert offload planner credits that repack to the resident experts, 576 more experts a rank on the TP4 serve
KV cache and prefix cache
waiting requests are admitted while a long prefill runs instead of after it ends
KV cache usage no longer counts blocks lent to expert copies as used, they come back on demand and read as free
a prefix cache fill figure joins the engine log and the metrics endpoint as vllm:prefix_cache_fill_perc, the share of GPU plus host block capacity holding reusable cached prefixes
with expert offload on the GPU prefix cache hit rate reads zero by design, the host tier serves the hits, read the external prefix cache hit rate
everything listed under :dev (29.09.3) below is in this image
Fixed major TP2 corruption entering mamba state impacting attention
recovered and accounted for every piece of memory vLLM uses, several allocations that were not counted before are now counted
turbo
the whole Qwen4 decode step runs as one persistent HIP launch, from routing and top k through the expert GEMMs, hyperconnection norms and all reduce to the head, sampler and MTP verify step, with no per operation round trips
the step is planned in Rust, which lays out the stages, barriers, workgroup roles and buffers into a descriptor table the HIP launch replays
each configuration of TP size, MTP, PLE mode, expert offload and draft head form gets its own library with the model compiled in
CLAV_Q4_TURBO=1 is new and enabled for the Qwen4 architecture at TP2 and TP4, with and without expert offload
turbo drafts in the launch only with the reduced draft head, so set CLAV_MTP_REDUCED_HEAD=1 for the full turbo decode rate on English primary workloads
this is turbo phase 1, the initial cut, turbo phase 2 will add more optimization to speed up both prefill and decode across all configurations
expert offload is automatic, no flags needed
the expert route is xmove, every expert one record of weights and scales in a pinned host hangar or a device gate, placed by the plan and moved over SDMA by a per expert traffic count each decode step
xmove serves MXFP4, block fp8 and symmetric int4 W4A16 experts, the W4A16 record covers GPTQ and AutoRound exports such as the Intel and Swift 1.5 Qwen3.8-Flash-Next checkpoints
when experts land on the host and no KV offload size or connector is given, the plan turns on the simple KV connector and sizes its host space to one full context request of KV blocks across all ranks, so device blocks stay free for expert copies
--kv-offloading-size overrides that space with more or less, for example --kv-offloading-size 6
more offload space keeps speed up longer under serve pressure, the recommended minimum holds 2 full context lengths
MTP
CLAV_MTP_REDUCED_HEAD now defaults to 0, the full draft vocabulary, and is no longer implied by CLAV_MTP_ORIGINAL=0
set CLAV_MTP_REDUCED_HEAD=1 for English primary workloads across chat, coding, entity extraction and general use, it gives a notable decode uplift and is what lets turbo draft in the launch instead of the speculator
the serve logs the full head choice at startup with that recommendation
CLAV_MTP_MXFP4_VERIFY=1, CLAV_MTP_W4_REPACK=1 and CLAV_MTP_W4_REFIT=1 stay on by default and raise the overall decode rate
CLAV_MTP_ORIGINAL=1 turns every MTP optimization off
scheduler
the Rust scheduler serves async scheduling configurations only and does not degrade to the Python scheduler, a configuration it cannot serve stops the boot with the reason in the log, and the Python scheduler runs only when CLAV_RUST_SCHEDULER=0 asks for it
the Rust and Python paths keep functionality parity, checked by parity tests against the Python classes, with some fixes applied on the Rust path
the refusal list names itself in the error, among them async scheduling off, a custom scheduler class, LoRA, pipeline parallelism, a non FCFS policy, pooling or encoder decoder models, KV cache events or metrics, and any KV connector other than the simple CPU offload connector
the Rust pieces under the scheduler, the KV cache manager and coordinator, the block pool, the attention and GDN managers, the request object and queues, the encoder cache and the KV offload manager, each have a CLAV_RUST_=0 switch for tests
KV cache size no longer slows decode with expert offload active, before this the slowdown grew as KV pressure grew
when experts are offloaded, the memory space is locked to the configured limit at the first request, and a new custom memory manager controls the activation space
the risk of an out of memory error at runtime should be nearly eliminated
fixed the vLLM idle spinner that never slept and held 4 CPU threads at 100% at boost clock, those threads now sit at no load
on a 9950X idle power drops from 100 W to 40 W, and inference load power drops from 170 W to 90 to 120 W
fixed a race in QSA sparse attention where a workgroup holding several items could overwrite a tile still being read, batches of 17 to 64 rows were affected
DSV4.1 1017 compatibility added for the exl3 checkpoint, using the original engrams from disk, as an initial test only, speed is horrible
--ple-nvme
it is set automatically when required
it adds nearly no overhead, except at TP4 with all experts on the GPUs, where it costs 4% prefill and 2% decode on a DRAMless PCIe 4.0 x4 NVMe drive
it runs at the speed of the SSD serving it, a PCIe 5.0 x4 drive should make the penalty near 0%
TP4 turbo result
decode weighted combined 271.0 t/s
decode raw combined across all 8 domains, unweighted, 278.1 t/s
prefill 9,467 t/s at 2K and 9,076 t/s at 64K
ITL cadence 10.0 ms at p50 and 11.0 ms at p99
TP2 turbo result
decode weighted combined 174.2 t/s
decode raw combined across all 8 domains, unweighted, 180.3 t/s