# NVFP4 Quantization

Native NVFP4 reinforcement learning starts with normal reward gains, then develops abnormal entropy, a widening train–sampling log-probability gap and falling reward late in training. miniTE separates quantization policy, linear layers and compute backends, and adds per-token scaling, contiguous layouts and amax reuse; on 8×B300, both miniTE paths with BF16 backward reach roughly 1.33–1.37× paired speedup over BF16 on complete learner updates, while a separate fixed-output serving test reaches about 1.64× throughput. These learning and system results come from different configurations and do not yet establish an end-to-end RL speedup.

Native NVFP4 reinforcement learning starts with normal reward gains, then develops abnormal entropy, a widening train–sampling log-probability gap and falling reward late in training. miniTE separates quantization policy, linear layers and compute backends, and adds per-token scaling, contiguous layouts and amax reuse; on 8×B300, both miniTE paths with BF16 backward reach roughly 1.33–1.37× paired speedup over BF16 on complete learner updates, while a separate fixed-output serving test reaches about 1.64× throughput. These learning and system results come from different configurations and do not yet establish an end-to-end RL speedup.

Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model, and SGLang and Miles added inference and reinforcement-learning training backends on launch day. Its text backbone has 816B parameters with about 42B active per token, accepting text, image, video and audio with a 1M-token context. On eight B300 GPUs with NVFP4 weights, an OpenHands multi-turn agentic coding replay reaches 80.3K total tokens/s per GPU at 61 tokens/s per user at concurrency 128.

Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model, and SGLang and Miles added inference and reinforcement-learning training backends on launch day. Its text backbone has 816B parameters with about 42B active per token, accepting text, image, video and audio with a 1M-token context. On eight B300 GPUs with NVFP4 weights, an OpenHands multi-turn agentic coding replay reaches 80.3K total tokens/s per GPU at 61 tokens/s per user at concurrency 128.

NVIDIA released Nemotron 3.5 Super VL, an open multimodal model that takes text, images, documents and video and returns text for always-on agents that read screens, documents and tools. SGLang supports deployment from day one, serving it with continuous batching, prefix caching and speculative decoding. The mixture-of-experts model has 120 billion total parameters and activates 12 billion per token, with built-in multi-token prediction; it reaches 243.9 tok/s on a single RTX 6000 PRO.

NVIDIA released Nemotron 3.5 Super VL, an open multimodal model that takes text, images, documents and video and returns text for always-on agents that read screens, documents and tools. SGLang supports deployment from day one, serving it with continuous batching, prefix caching and speculative decoding. The mixture-of-experts model has 120 billion total parameters and activates 12 billion per token, with built-in multi-token prediction; it reaches 243.9 tok/s on a single RTX 6000 PRO.

NVIDIA released Nemotron 3.5 Super VL, an open multimodal model for long-running agents that folds document, image, and video understanding together with reasoning, instruction following, and tool use in one model, reducing the need to deploy separate text and vision models. It uses a 120-billion-parameter mixture-of-experts design with 12 billion active parameters, combined with a hybrid Mamba-Transformer architecture, multi-token prediction, and NVFP4 quantization, and can be post-trained with NVIDIA NeMo on an organization's own data, tools, policies, and workflows. The stated comparison is up to 5x faster throughput on agentic tasks than other open models in its size category on a single GPU, and 4x faster task completion.

NVIDIA released Nemotron 3.5 Super VL, an open multimodal model for long-running agents that folds document, image, and video understanding together with reasoning, instruction following, and tool use in one model, reducing the need to deploy separate text and vision models. It uses a 120-billion-parameter mixture-of-experts design with 12 billion active parameters, combined with a hybrid Mamba-Transformer architecture, multi-token prediction, and NVFP4 quantization, and can be post-trained with NVIDIA NeMo on an organization's own data, tools, policies, and workflows. The stated comparison is up to 5x faster throughput on agentic tasks than other open models in its size category on a single GPU, and 4x faster task completion.

The work introduces OSFP4: by analyzing a multiplicatively dithered FP4 quantizer it obtains a differentiable approximation of the matrix-product quantization error, then jointly optimizes a diagonal smoothing matrix and weight block scales per linear projection, with separate losses for RTN and GPTQ-style SIC rounding; on Llama-3.1-8B-Instruct the W4A4 configuration attains the highest average accuracy among the evaluated methods while retaining roughly 94-97% of vendor NVFP4 prefill throughput.

The work introduces OSFP4: by analyzing a multiplicatively dithered FP4 quantizer it obtains a differentiable approximation of the matrix-product quantization error, then jointly optimizes a diagonal smoothing matrix and weight block scales per linear projection, with separate losses for RTN and GPTQ-style SIC rounding; on Llama-3.1-8B-Instruct the W4A4 configuration attains the highest average accuracy among the evaluated methods while retaining roughly 94-97% of vendor NVFP4 prefill throughput.

The authors present SoloQ, which uses a K-RPBH rotation to map weights and activations into a predictable near-Gaussian basis plus a scale-only bias correction for calibration-free 4-bit quantization, and quantizes the KV cache at block commit for block-diffusion models; across LLaDA-Base, LLaDA-1.5, Dream-7B, Fast-dLLM v2 and Nemotron-Labs-Diffusion, SoloQ-C and SoloQ-N retain accuracy under W4A4 and W4A4KV4 and exceed calibration-based baselines such as DLLMQuant++, STaR-Quant and FAIR-Calib on average, with the NVFP4 path also cutting peak memory and speeding up end-to-end inference.

The authors present SoloQ, which uses a K-RPBH rotation to map weights and activations into a predictable near-Gaussian basis plus a scale-only bias correction for calibration-free 4-bit quantization, and quantizes the KV cache at block commit for block-diffusion models; across LLaDA-Base, LLaDA-1.5, Dream-7B, Fast-dLLM v2 and Nemotron-Labs-Diffusion, SoloQ-C and SoloQ-N retain accuracy under W4A4 and W4A4KV4 and exceed calibration-based baselines such as DLLMQuant++, STaR-Quant and FAIR-Calib on average, with the NVFP4 path also cutting peak memory and speeding up end-to-end inference.