verdictai/deepseek-v41-flash-trellismx repository overview

⁠DeepSeek-V4.1-Flash TrellisMX serving image

Serving image for the TrellisMX builds of DeepSeek's DeepSeek-V4.1-Flash-1017 on RTX PRO 6000 Blackwell 96 GB cards (SM120), with DSpark speculative decoding and full CUDA graphs. The model weights are not in the image; they come from the Hugging Face repositories below.

Digests: k2-fp8-tp2-20261019 = sha256:73a113d91c943a68da5774a61ece057c224f4b476c398b8cd15f8d9450ca82c2, k3-fp8-tp3-20261019 = sha256:58aeb5a628920af36e1b0721ca07187636092e95f867cadbec6bb69f66362be3, k4-fp8-tp4-20261019 = sha256:d35737d826e0d2a00d7f603e5a76f220103e95312c78dd5580ef862c322a5bfa, k2mix-fp8-tp2-test-20261020 = sha256:a72e633c8779c6903c4039dc2397e8a55db4a1669f422c9f6516de97e04c0c60. All four are built on ghcr.io/local-inference-lab/vllm@sha256:f5f9efe1b10ec7b853a495e66b7dbf15ff54f62741e444dea0b44e16736a10a9.

⁠Run it

Download the checkpoint from Hugging Face and use the serving/docker-compose.yml and serving/serve.sh from that repository: serve.sh reads the checkpoint's manifest, picks the matching image and tensor-parallel size, checks the free host RAM and starts the container. The model card has the step-by-step instructions and serving/SERVING.md every setting.

⁠Licenses

The image contains third-party software under its own licenses: the vLLM image by Local Inference Lab and Luke Alonso (a fork of vLLM, Apache-2.0) and the projects it bundles (FlashInfer, LMCache, InstantTensor, xgrammar, fastokens), and the b12x kernels (Apache-2.0). The checkpoints are released under the MIT License, the license of the base model.

⁠Credits

• Compute: Lambda. The checkpoints were encoded and measured on an 8x NVIDIA B200 node that Lambda⁠ donated for this work. Our sincere thanks to Lambda (@LambdaAPI⁠) for supporting independent open-model research.
• Base model. DeepSeek-V4.1-Flash-1017 by DeepSeek (deepseek-ai/DeepSeek-V4.1-Flash-1017), MIT License.
• Image: Local Inference Lab and Luke Alonso. Built on the vLLM image by Local Inference Lab and Luke Alonso (ghcr.io/local-inference-lab/vllm, https://github.com/local-inference-lab⁠), a fork of vLLM (Apache-2.0, vLLM contributors).
• Kernels: b12x by Luke Alonso. The MoE, attention and GEMM kernels come from b12x (https://github.com/lukealonso/b12x⁠, mirrored at https://github.com/local-inference-lab/b12x⁠) by Luke Alonso, with Martin Vit and other contributors (Apache-2.0).
• CSF encoder: Local Inference Lab. The lossless compressed scale format (CSF) the expert scales are stored in, and its encoder and decoder, are Local Inference Lab's work.
• TrellisMX: QTIP and EXL3. TrellisMX, Brandon M. Music's trellis-coded format, was inspired by QTIP (Albert Tseng, Qingyao Sun, David Hou and Christopher De Sa, NeurIPS 2024) and by ExLlamaV3 and its EXL3 format by turboderp and contributors (MIT License).
• Codec changes: QSRT and Luke Alonso. Decoding the trellis directly into FP8 (E4M3) tensor-core operands was inspired by Luke Alonso's KQuant/QSRT work and the b12x W4A8 trellis path.