vLLM runs an 816B sparse model on day one: 1M context at a quarter of the KV cache, over 960 tokens/sec on eight TPU v7 chips

Muse Spark Open can be deployed on vLLM the day it is released, with prebuilt images for Nvidia and AMD and weights in both BF16 and NVFP4, plus MXFP4 weights from AMD. By vLLM's own note, the model reaches the nightly PyPI wheel only after the code is reviewed and merged, which usually takes about a week, so images and wheel do not land at the same time. Support spans two vendors: Nvidia covers tensor and pipeline parallelism plus EP, TEP, DEP and PD disaggregation, AMD covers tensor and pipeline parallelism, and TPU has initial support on TorchTPU.

Long-context memory is cut by letting most layers see only a recent window of tokens. Full attention would keep keys and values for every layer across 1M tokens, whereas 46 of the 62 layers here use sliding-window attention holding only the most recent 2048 tokens, and 16 layers see the whole context. With a repeating cycle of four anchored to the end of the stack so the final layer is always full attention, the cache at 1M tokens drops to just over a quarter of what all-full attention needs, which is what makes that context length affordable.

Inference speedup comes from a purpose-trained speculative draft model available on both CUDA and ROCm. Speculative decoding traditionally gives diminishing returns at large batch sizes, and DSpark uses confidence-based scheduling to sustain speedups there. DSpark uses block diffusion to propose up to seven draft tokens in one pass and reuses the sliding-window attention to keep drafting light, tested on the SPEED bench.

vLLM also serves Muse Spark Open as a reinforcement-learning rollout engine, with Verl FSDP2 as the trainer. The server runs the original NVFP4 checkpoint with a LoRA adapter on top, while the trainer runs on a BF16 dequantization of the same weights and updates only the adapter. The adapter is rank 32 with alpha 64, covering attention, the MoE latent projections and all routed experts, and the rollout's MoE routing is replayed; GRPO on DAPO-Math-17K ran 150 steps on 16 GB300 GPUs, lifting AIME-2024 mean@8 from 0.56 to 0.85 while the k3 KL between trainer and vLLM log-probabilities held near 2×10⁻⁴.

This report is for researchers and engineers who need to deploy or post-train large sparse models, covering inference support on Nvidia, AMD and TPU and a LoRA reinforcement-learning loop with vLLM as the rollout engine. The TPU path requires contacting vLLM or Google for code access, and the model reaches the nightly PyPI wheel only after code review and merge.

The TPU throughput figures correspond to megakernels with DSpark speculative decoding at acceptance length 6; without speculative decoding, four and eight chips deliver over 270 and 350 tokens/sec respectively. The reinforcement-learning result comes from 150 GRPO steps on DAPO-Math-17K, sampling 8 responses for each of 32 prompts per step with up to 4K tokens per response. A video's audio track is not processed by the audio encoder, so video and extracted audio must be sent as two separate content parts in the same request.