HUB
Sign inSign up

tcclaviger/vllm

By tcclaviger

•Updated 10 days ago

Image
16

50K+

tcclaviger/vllm repository overview

⁠tcclaviger/vllm - ROCm/RDNA4 vLLM with native HIP kernels, baked-in quantizers + tuner

vLLM 0.30 tree on ROCm 10.0, torch 2.11, Python 3.14, built for gfx1201 (Radeon R9700). With no tool flag the image runs vllm serve "$@", so it drops into an existing vLLM deployment unchanged.

Docs, examples and serving recipes: https://www.cogferra.com/vllmdocs⁠

⁠Tags

All tags ship libr4d, the RDNA4 HIP kernel library: https://codeberg.org/StillDeadcode/libr4d⁠. Unlisted features may be present as drafts.

⁠Credits

  • StillDeadcode - libr4d
  • Davetha - int4 draft lm_head, LRU expert cache behind expert offload
  • Pat Carter - RDNA4 GDN kernels, rdna4-vllm⁠
  • Dylhun - KVA and bug hunts
  • Code 000 - refinements
  • ggz14 - kernels
  • Everyone on Launch80 who contributed ideas, suggestions and testing

⁠Change log

⁠:latest (29.09.11) benchmark https://www.cogferra.com/resultviewer?id=168⁠
  • turbo is on by default for the Qwen4 architecture at TP2 and TP4, with and without expert offload, CLAV_Q4_TURBO=0 turns it off and runs the eager path
  • turbo on block fp8 checkpoints
    • turbo now serves the block fp8 checkpoints of Qwen3.8-Flash-Next, the expert GEMMs run inside the persistent launch and are bit identical to the eager fp8 path
    • the expert format is read from the checkpoint at startup, MXFP4 and block fp8 each have their own launch libraries and the serve picks the right one with no flag
    • the fp8 libraries read the experts through xmove, so an fp8 checkpoint whose experts do not fit the cards serves from host memory with the launch on
    • an fp8 checkpoint whose attention projections are stored in bf16 gets them quantized to fp8 block 128 at load, CLAV_PROJ_FP8_REPACK=0 keeps them bf16 and the launch off
    • the expert offload planner credits that repack to the resident experts, 576 more experts a rank on the TP4 serve
  • KV cache and prefix cache
    • waiting requests are admitted while a long prefill runs instead of after it ends
    • KV cache usage no longer counts blocks lent to expert copies as used, they come back on demand and read as free
    • a prefix cache fill figure joins the engine log and the metrics endpoint as vllm:prefix_cache_fill_perc, the share of GPU plus host block capacity holding reusable cached prefixes
    • with expert offload on the GPU prefix cache hit rate reads zero by design, the host tier serves the hits, read the external prefix cache hit rate
  • everything listed under :dev (29.09.3) below is in this image
⁠:dev (29.09.3) benchmark https://www.cogferra.com/resultviewer?id=162⁠
  • Fixed major TP2 corruption entering mamba state impacting attention
  • recovered and accounted for every piece of memory vLLM uses, several allocations that were not counted before are now counted
  • turbo
    • the whole Qwen4 decode step runs as one persistent HIP launch, from routing and top k through the expert GEMMs, hyperconnection norms and all reduce to the head, sampler and MTP verify step, with no per operation round trips
    • the step is planned in Rust, which lays out the stages, barriers, workgroup roles and buffers into a descriptor table the HIP launch replays
    • each configuration of TP size, MTP, PLE mode, expert offload and draft head form gets its own library with the model compiled in
    • CLAV_Q4_TURBO=1 is new and enabled for the Qwen4 architecture at TP2 and TP4, with and without expert offload
    • turbo drafts in the launch only with the reduced draft head, so set CLAV_MTP_REDUCED_HEAD=1 for the full turbo decode rate on English primary workloads
    • this is turbo phase 1, the initial cut, turbo phase 2 will add more optimization to speed up both prefill and decode across all configurations
  • expert offload is automatic, no flags needed
    • the expert route is xmove, every expert one record of weights and scales in a pinned host hangar or a device gate, placed by the plan and moved over SDMA by a per expert traffic count each decode step
    • xmove serves MXFP4, block fp8 and symmetric int4 W4A16 experts, the W4A16 record covers GPTQ and AutoRound exports such as the Intel and Swift 1.5 Qwen3.8-Flash-Next checkpoints
    • when experts land on the host and no KV offload size or connector is given, the plan turns on the simple KV connector and sizes its host space to one full context request of KV blocks across all ranks, so device blocks stay free for expert copies
    • --kv-offloading-size overrides that space with more or less, for example --kv-offloading-size 6
    • more offload space keeps speed up longer under serve pressure, the recommended minimum holds 2 full context lengths
  • MTP
    • CLAV_MTP_REDUCED_HEAD now defaults to 0, the full draft vocabulary, and is no longer implied by CLAV_MTP_ORIGINAL=0
    • set CLAV_MTP_REDUCED_HEAD=1 for English primary workloads across chat, coding, entity extraction and general use, it gives a notable decode uplift and is what lets turbo draft in the launch instead of the speculator
    • the serve logs the full head choice at startup with that recommendation
    • CLAV_MTP_MXFP4_VERIFY=1, CLAV_MTP_W4_REPACK=1 and CLAV_MTP_W4_REFIT=1 stay on by default and raise the overall decode rate
    • CLAV_MTP_ORIGINAL=1 turns every MTP optimization off
  • scheduler
    • the Rust scheduler serves async scheduling configurations only and does not degrade to the Python scheduler, a configuration it cannot serve stops the boot with the reason in the log, and the Python scheduler runs only when CLAV_RUST_SCHEDULER=0 asks for it
    • the Rust and Python paths keep functionality parity, checked by parity tests against the Python classes, with some fixes applied on the Rust path
    • the refusal list names itself in the error, among them async scheduling off, a custom scheduler class, LoRA, pipeline parallelism, a non FCFS policy, pooling or encoder decoder models, KV cache events or metrics, and any KV connector other than the simple CPU offload connector
    • the Rust pieces under the scheduler, the KV cache manager and coordinator, the block pool, the attention and GDN managers, the request object and queues, the encoder cache and the KV offload manager, each have a CLAV_RUST_=0 switch for tests
  • KV cache size no longer slows decode with expert offload active, before this the slowdown grew as KV pressure grew
  • when experts are offloaded, the memory space is locked to the configured limit at the first request, and a new custom memory manager controls the activation space
  • the risk of an out of memory error at runtime should be nearly eliminated
  • fixed the vLLM idle spinner that never slept and held 4 CPU threads at 100% at boost clock, those threads now sit at no load
    • on a 9950X idle power drops from 100 W to 40 W, and inference load power drops from 170 W to 90 to 120 W
  • fixed a race in QSA sparse attention where a workgroup holding several items could overwrite a tile still being read, batches of 17 to 64 rows were affected
  • DSV4.1 1017 compatibility added for the exl3 checkpoint, using the original engrams from disk, as an initial test only, speed is horrible
  • --ple-nvme
    • it is set automatically when required
    • it adds nearly no overhead, except at TP4 with all experts on the GPUs, where it costs 4% prefill and 2% decode on a DRAMless PCIe 4.0 x4 NVMe drive
    • it runs at the speed of the SSD serving it, a PCIe 5.0 x4 drive should make the penalty near 0%
  • TP4 turbo result
    • decode weighted combined 271.0 t/s
    • decode raw combined across all 8 domains, unweighted, 278.1 t/s
    • prefill 9,467 t/s at 2K and 9,076 t/s at 64K
    • ITL cadence 10.0 ms at p50 and 11.0 ms at p99
  • TP2 turbo result
    • decode weighted combined 174.2 t/s
    • decode raw combined across all 8 domains, unweighted, 180.3 t/s
    • prefill 5,464 t/s at 2K and 6,295 t/s at 64K
    • ITL cadence 14.9 ms at p50 and 20.2 ms at p99

Tag summary

Content type

Image

Digest

sha256:8a93d50f5…

Size

4 GB

Last updated

10 days ago

docker pull tcclaviger/vllm