# Exact Quantile Balancing

In sparse mixture-of-experts models, a router that favors a few experts starves the rest of training signal and makes devices wait under expert parallelism. Kolibri pre-training pairs Exact Quantile Balancing (EQB), which sets routing biases from exact global-batch quantiles via two-pass radix selection, with bounded Load-Error Injection (LEI), which adds each microbatch's load error directly to router gradients. On a 7.5B-parameter proxy trained for 100B tokens, EQB cut global maximal violation from 0.92 to 0.74, and adding bounded LEI cut local violation from 5.38 to 3.52 at a mean BPB cost from 0.6570 to 0.6720.

In sparse mixture-of-experts models, a router that favors a few experts starves the rest of training signal and makes devices wait under expert parallelism. Kolibri pre-training pairs Exact Quantile Balancing (EQB), which sets routing biases from exact global-batch quantiles via two-pass radix selection, with bounded Load-Error Injection (LEI), which adds each microbatch's load error directly to router gradients. On a 7.5B-parameter proxy trained for 100B tokens, EQB cut global maximal violation from 0.92 to 0.74, and adding bounded LEI cut local violation from 5.38 to 3.52 at a mean BPB cost from 0.6570 to 0.6720.