krisbailey/qwen3.8-flash-next-nvfp4-dgx-spark repository overview

⁠Qwen3.8-Flash-Next-NVFP4 on GB10

Serves primitive-ai/Qwen3.8-Flash-Next-NVFP4⁠ on a DGX Spark (GB10). Point it at an empty cache volume and it downloads the model, builds what it needs, and starts serving. No manual checkpoint surgery.

⁠Requirements

• A GB10 (compute capability 12.1). This won't run on anything else — the kernels in here are compiled for this specific chip, not CUDA GPUs in general.
• Driver ≥ 580.95.05, CUDA 13.0.
• Docker + the NVIDIA Container Toolkit (--gpus all needs to work).
• ~174 GiB free disk for the model.
• Headroom to load a 78 GiB checkpoint. GB10 is unified memory — the weights come out of the same pool the OS calls "available RAM," not separate VRAM, so don't expect to run this alongside something else memory-hungry and have both be happy.

⁠Running it

docker run -d --name qwen38-flash-next \
  --gpus all --ipc=host -p 8000:8000 \
  -v qwen-cache:/root/.cache \
  <your-registry>/qwen3.8-flash-next-nvfp4-dgx-spark:latest

Mount the whole /root/.cache, not just the Hugging Face cache directory. GPU kernel autotuning (FlashInfer, torch.compile) also lands under /root/.cache, and if you only persist the HF side you'll re-pay the autotuning cost — several minutes — on every restart, not just the first one.

First boot is slow and that's normal: download the model if it's not cached, load 78 GiB of weights off disk, then (first boot on a fresh cache volume only) autotune the GPU kernels. Could be twenty minutes, could be longer depending on your connection. There's a HEALTHCHECK on /health with a 40-minute start period, or just poll it yourself:

docker inspect --format '{{.State.Health.Status}}' qwen38-flash-next

⁠What this actually is

This is upstream vllm/vllm-openai:qwen38-flash-next with a handful of file overlays on top — nothing hidden. docker history shows every layer, each one a plain-text file a few KB to ~1 MB.

The GPU this targets (GB10 / SM121) has a real cuBLAS problem: its BF16 tensor-core GEMM path is broken under CUDA 13.0. The fix that actually matters here is enabling vLLM's CUTLASS "skinny GEMM" kernel for this chip — it was gated to a different Blackwell variant (SM103) and just needed the gate widened, plus some shape tuning for this model's dimensions at TP=1. That's the patch doing the real work: it's what the decode-step throughput depends on.

On top of that, the MTP draft head can score a curated 16,000-token subset of the 248,320-token vocabulary instead of the whole thing (shortlists/lowcore_16k.json). Worth having, but don't oversell it to yourself: it's a modest single-stream win (roughly 4-10%, and only once it's routed through the skinny kernel too — otherwise it can lose to the full head). It does close to nothing for saturated/concurrent throughput. The full target model verifies every token it proposes either way, so none of this can change what comes out, only how fast.

Also in here: a PLE (position-learning embedding) table streaming setup that reads the model's ~102 GiB ngram table straight off disk via a byte-offset index instead of loading it into memory, and a one-line chat-template fix so an out-of-range reasoning_effort degrades instead of hard-erroring.

⁠Config

Everything below is an env var, or pass it as an extra flag to docker run (it wins over this image's own default since it comes after):

⁠If something's wrong

• "expected a GB10 GPU" — the kernels here are compiled for SM121 specifically. Right error, wrong hardware.
• Nothing happening for ages — check docker logs -f. Shard-loading percentages or kernel autotune progress mean it's working, just slow. See the timing note above.
• "Available RAM" looks low during weight loading — expected, see Requirements. Unified memory means the model itself eats into that number as it loads.
• Every restart is slow, not just the first one — you probably mounted /root/.cache/huggingface instead of all of /root/.cache. The autotuning caches live outside the HF path.