tcclaviger/vllm - ROCm/RDNA4 vLLM with native HIP kernels, baked-in quantizers + tuner
vLLM 0.30 tree on ROCm 10.0, torch 2.11, Python 3.14, built for gfx1201 (Radeon R9700). With no tool flag the image runs vllm serve "$@", so it drops into an existing vLLM deployment unchanged.
TP4 is the intended serve setup, smaller TP sizes are supplementary and may or may not work correctly
source stripped, comments stripped, reasons
turbo
decode launches have a wht6 all reduce option at TP4 with R4D_AR_WIRE=wht6, the default stays ti8
the exact one shot still serves up to 12 rows at TP4 and up to 32 rows at TP2, the quantized wire runs above that
R4D_AR_QUANT=0 forces the exact wire and turbo declines the decode shapes above the exact band
TP2 is now served by turbo with the exact and wht6 one shot wires if expert offloading is not active, which it is by default at TP2, turbo cannot serve with expert offloading so when the plan keeps experts on the host the serve logs it and boots on today's path with CLAV_Q4_TURBO ignored
MTP
CLAV_MTP_ORIGINAL=1 overrides all MTP optimizations and saves memory while keeping full multilingual support, 0 enables the MTP optimizations and reduces MTP efficacy outside english and code work
the draft attention projections load as fp8 and the int4 draft head keeps the least squares scale refit
CLAV_MTP_REDUCED_HEAD=1 cuts the speculator lm_head size
CLAV_MTP_W4_REPACK=1 and CLAV_MTP_W4_REFIT=1 are w4a8 repacks of the MTP experts
CLAV_MTP_MXFP4_VERIFY=1 is the verification head and matched at topk 16 in 100% of tested cases
the head, repack and verify options take a small amount of memory to run but make TP4 a net gain instead of a net loss
index sharing across the draft steps is on with index_share_for_mtp_iteration true in the speculative config, as in method mtp, num_speculative_tokens 3, draft_sample_method probabilistic, index_share_for_mtp_iteration true
fixed mamba alignment at a block transition when the MTP tokens reach 2 or more beyond the end of the mamba block
hyperconnection and attention
fused HC and the fused all reduce paths run by default in BF16 HC mode
CLAV_HC_FP8_REPACK=1 is now the default, a significant decode uplift at the cost of occasional small top k 1 flips
QSA v2 sparse attention is now the default, a significant prefill uplift at the cost of occasional small top k 1 flips, CLAV_QSA_V2=0 opts back
expert offload and PLE
expert offload fully reworked as a single copy cache that plans itself from the model and is now the default
no extra copies of experts are kept anymore, so the offload footprint in system RAM is much smaller
kv cache sizing is exact in expert offload mode, so no VRAM is wasted anymore
the --ple-nvme path is fully reworked and now runs at drive access speed, it no longer hampers performance meaningfully
ple-nvme enables itself at startup on systems that cannot fit the offloaded experts and the PLE table in host memory
expert cache metrics in the engine log beside the spec decode line every stats interval
Expert cache: Hit rate: 94.4%, Expert rotations: 6600, Bandwidth: 1.73 GB/s, UVA expert reads: 14927, Direct expert read: 3.91 GB/s
kv prefix offload
VLLM_USE_SIMPLE_KV_OFFLOAD=1 in the environment turns it on
--kv-offloading-size 64 sizes the host RAM tier, the number is GiB total across all ranks
--kv-transfer-config adds the disk tier through its kv_connector_extra_config
kv_offload_backend disk selects the disk tier
disk_path is the file path the tier writes to, one file per rank
disk_capacity_bytes is the disk tier size in bytes, default 100 GiB
validated with PLE in RAM, PLE LRU cache and PLE NVMe direct on TP2 and TP4, on both the RAM and the disk tier
scheduler
the Rust scheduler is on by default and migrates the scheduler into Rust, CLAV_RUST_SCHEDULER=0 turns every Rust piece off
models
the Quark MXFP4 checkpoint of Qwen3.8-Flash-Next now serves, its quark mxfp4 layers are routed to the r4d mxfp4 kernels
the Intel AutoRound W4A16 int4 checkpoint of Qwen3.8-Flash-Next is validated
the Swift 1.5 Qwen3.8-Flash-Next AutoRound W4A16 checkpoint is validated
accuracy
many of the levers introduce small perturbations in the outputs, the most accurate serving is CLAV_HC_FAST=0, CLAV_MTP_ORIGINAL=1, probabilistic MTP off and CLAV_HC_FP8_REPACK=0