verdictai/muse-spark-trellismx repository overview

⁠Muse Spark TrellisMX serving image

Serving image for the TrellisMX builds of Meta's Muse Spark Open 816B-A42B on four RTX PRO 6000 Blackwell 96 GB cards (SM120) at tensor parallel 4, with DFlash2 speculative decoding, full CUDA graphs and image, video and audio input. The model weights are not in the image; they come from the gated Hugging Face repositories below.

⁠Run it

Request access to the checkpoint on Hugging Face, download it, and use the serving/docker-compose.yml and serving/serve.sh from that repository; its model card has the step-by-step instructions and the measured figures.

⁠Licenses

The image contains third-party software under its own licenses (listed below). The checkpoints are subject to the Muse Model License and the Muse Spark Open Usage Policy; the DFlash2 drafter shipped with them is under the NVIDIA Open Model Agreement.

⁠Credits

• Base model. Muse Spark Open by Meta (meta-models/Muse-Spark-Open-816B-A42B).
• Base image: Local Inference Lab and Luke Alonso. Built on the vLLM image by Local Inference Lab and Luke Alonso (ghcr.io/local-inference-lab/vllm, https://github.com/local-inference-lab⁠), a fork of vLLM (Apache-2.0, vLLM contributors), with the open-source projects it bundles, each under its own license: FlashInfer, LMCache, InstantTensor, xgrammar and fastokens.
• Kernels: b12x by Luke Alonso. The MoE, attention and GEMM kernels come from b12x (https://github.com/lukealonso/b12x⁠, mirrored at https://github.com/local-inference-lab/b12x⁠) by Luke Alonso, with Martin Vit and other contributors (Apache-2.0).
• CSF encoder: Local Inference Lab. The lossless compressed scale format (CSF) for the expert scales, with its encoder and decoder, is Local Inference Lab's work, shipped in Local Inference Lab's FlashInfer.
• TrellisMX: QTIP and EXL3. TrellisMX, Brandon M. Music's trellis-coded format, was inspired by QTIP (Albert Tseng, Qingyao Sun, David Hou and Christopher De Sa, NeurIPS 2024) and by ExLlamaV3 and its EXL3 format by turboderp and contributors (https://github.com/turboderp-org/exllamav3⁠, MIT License).
• Codec changes: QSRT and Luke Alonso. Decoding the trellis directly into FP8 and FP4 tensor-core operands was inspired by Luke Alonso's KQuant/QSRT work and the b12x W4A8 trellis path.
• Methods. GPTQ (Frantar, Ashkboos, Hoefler and Alistarh, ICLR 2023); QuIP (Chee, Cai, Kuleshov and De Sa, NeurIPS 2023); QuIP# (Tseng, Chee, Sun, Kuleshov and De Sa, ICML 2024); QuaRot (Ashkboos et al., NeurIPS 2024); SpinQuant (Liu et al., 2024); the OCP Microscaling (MX) Formats Specification (Rouhani et al., 2023).
• Drafter: NVIDIA. DFlash2 drafter for Muse Spark (NVIDIA Open Model Agreement), included in the checkpoint repos.
• Evaluation data. The KLD figures on the model cards use public datasets credited, with their licenses, on those cards: NVIDIA Nemotron SFT, pretraining and VLM sets, Nebius SWE-agent-trajectories, LibriSpeech and FLEURS.