vLLM runs Mistral Large 4 on day one: NVFP4 decodes 1.08–1.19x faster than FP8 with about 1.5x the KV cache
Synopsis
vLLM added same-day inference support for Mistral Large 4, a 1-trillion-parameter sparse mixture-of-experts multimodal model with 49 billion parameters active per token. The work spans breakable CUDA graphs, hand-written fusions, decode and prefill context parallelism, DSpark speculative decoding, NIXL disaggregated serving and host-memory KV offload. On TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 and holds about 1.5x the KV cache, while staying within 2.1 points of FP8 on multimodal and long-context evaluations.
Interpretation
vLLM can serve Mistral Large 4 on the day it ships, across Nvidia GPUs, AMD Instinct MI350 Series and TPU v7x. Before this, vLLM had no model package, layer spec or speculative path for this architecture, and a model file alone was not enough to deploy it. Support ships in the vllm/vllm-openai:mistral-large-4 image with mistral-common 1.13, with commands for FP8 on eight GPUs and NVFP4 on four.
The model has 72 layers of mixture-of-experts with 128 routed experts taking the top 4 plus one shared expert, hidden size 6144, 1 trillion parameters total and 49 billion active per token. Attention is MLA with KV LoRA rank 512 and RoPE dimension 64, plus a DSA sparse indexer that selects the top 2048 keys per query. Nine of the layers from 39 to 71, one every four, read layer 35's latent cache, so 63 of the 72 layers own a KV cache.
The NVFP4 checkpoint does not trade quality for speed: on multimodal, long-context, agentic coding, business automation and cyber evaluations it stays within 2.1 points of FP8 on every task. The mean difference is −0.3 points, and mean completion tokens stay within about 4% of FP8. At TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 without speculation and 1.18–1.22x with DSpark, while holding about 1.5x the KV cache.
The DSpark speculative drafter ships inside the checkpoint as three sliding-window MLA draft layers at depth 3. It proposes all draft positions of a block in one forward pass, conditioned on hidden states tapped from target layers 1, 18, 36, 54 and 71. On TP8 with 200 real prompts and greedy decoding, FP8 c1 per-token time falls from 8.84 ms to 3.50 ms and NVFP4 from 8.21 ms to 2.95 ms.
Perspective
The report covers vLLM's inference support and measured configurations for Mistral Large 4, including FP8 and NVFP4 on Nvidia GPUs, the ROCm path on AMD Instinct MI350 Series, and a validated 16-chip TPU v7x configuration; the numbers correspond to the specific GPU counts, concurrencies and input lengths listed.
DCP trades per-step latency for KV capacity: on the same 8-GPU GB300 NVFP4 deployment, DCP8 decodes about 16–28% slower per token than TP8. Under DCP8 each 64-token DSpark draft block holds a block ID whose target pages cover 512 tokens, so at 64k context capacity falls to about 0.69x of plain DCP8. On a single-node tensor-parallel group, HiSparse also requires VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm to avoid a CUDA launch failure.