Native NVFP4 RL destabilizes late in training: miniTE lifts full learner updates to about 1.33–1.37× BF16
Synopsis
Native NVFP4 reinforcement learning starts with normal reward gains, then develops abnormal entropy, a widening train–sampling log-probability gap and falling reward late in training. miniTE separates quantization policy, linear layers and compute backends, and adds per-token scaling, contiguous layouts and amax reuse; on 8×B300, both miniTE paths with BF16 backward reach roughly 1.33–1.37× paired speedup over BF16 on complete learner updates, while a separate fixed-output serving test reaches about 1.64× throughput. These learning and system results come from different configurations and do not yet establish an end-to-end RL speedup.
Interpretation
Native NVFP4 RL becomes unstable late in training, and miniTE works on the numerical-execution side, reducing avoidable execution differences and lowering the cost of low-precision computation. In early NVFP4 runs reward rose normally before entropy became abnormal, the train–sampling log-probability gap grew and reward declined; replacing a single quantization operator, or confirming both sides used NVFP4, did not resolve all of these issues. GRPO on DAPO mathematics tasks with Qwen3-4B and Qwen3-30B-A3B showed the 4B model unstable after about 300 steps and the 30B mismatch growing rapidly after about 700 steps; in early configurations the dense NVFP4/NVFP4 mean absolute log-probability gap was 0.1142 against 0.0110 for BF16/BF16.
miniTE separates quantization policy, linear layers and compute backends so a developer can swap one compute path inside an existing training workflow and then check numerical correctness, learning behavior and runtime cost separately. Previously, changing scale granularity or a GEMM backend tended to require changes to layer wrappers, grouped layouts, communication and autograd, so each experiment revisited the surrounding logic. miniTE per-token dense quantization and grouped GEMM pass 28/28 bitwise equivalence checks, and fused layouts add no discrepancy across seven synthetic and real-activation cases; on 8×B300, miniTE CUTLASS with BF16 backward reaches 33,140 tokens/s at 1.369× paired speedup and miniTE per-token with BF16 backward reaches 32,517 tokens/s at 1.334×, while TE NVFP4 reaches 23,294 tokens/s at 0.959×.
Perspective
The demonstrated scope is Qwen3-30B-A3B learner updates on a single 8×B300 node (sequence length 4096, micro-batch 1, GBS 128), plus a separate eight-engine fixed-output serving test and one staged online-RL timing; module, learner, serving and online-RL tests each use a different timing scope.
Under the strictly aligned configuration the train–inference log-probability gap stays around 10⁻⁸, but that experiment changes several implementation choices together and is a single run; in the supplementary long run miniTE's five-task mean is 56.35%, still 2.49 points below BF16 and below native NVFP4 on IFEval. In the online RL run, weight conversion and synchronization are about 5.56× and 5.12× BF16, offsetting sampling gains, and whether combining the two tracks adds further gains remains to be tested.