MoE serving re-splits expert replicas every decode step: Relief's integer relaxation nears the per-step optimum, and an on-die engine lifts effective utilization to 99.5%

An expert-parallel MoE layer completes each decode step only when the most loaded GPU in its group finishes, leaving the rest of the reserved GPU time idle; Relief makes the replica-split decision once per layer per decode step instead of relying on minute-scale expert re-placement. Production engines currently pick among an expert's replicas by position modulo or at random between minute-scale EPLB re-placements, consulting no load signal; the exact per-step linear-program solver takes about 102 microseconds per solve, and on short layers the utilization it loses exceeds the balance it recovers. On GSM8K and WildChat decode routing from Qwen3-30B-A3B, OLMoE-1B-7B and Qwen1.5-MoE-A2.7B, scored as effective utilization relative to the per-step optimum, the deployed static placement reaches 66% to 72%, the GPU kernel 93%, and the on-die engine 99.5%.

The algorithm treats each replicated expert's token division between its two copies as a set of adjustable fractions and edits the per-GPU load table directly, moving load from the heavier side to the lighter one. The exact solvers obtain those fractions from a min-max linear program, whereas Relief performs the same balancing with integer addition, comparison and shift, and its working set is the load vector itself. Each round halves the load gap between the two ranks of a replica edge and writes the result back immediately for the next round to read; on torus placements from 8 to 64 GPUs, a few passes land within a fraction of a point of the per-step optimum.

The same relaxation is built three ways: a CUDA kernel on the serving GPU, a fixed-function silicon block beside the accelerator, and an FPGA card across PCIe, all returning bit-identical splits. The kernel needs only the GPU a deployment already owns, but its load table sits in shared memory behind a roughly 25-nanosecond load-to-use path; the on-die engine holds the same table in registers, so the dependent read becomes a wire delay. The on-die engine occupies 0.14 square millimeters of standard cells and about ten million transistors in ASAP7 7 nm, roughly 0.02% of a current accelerator by transistor count; a solve takes 0.75 microseconds at 16 GPUs and 0.79 at 64, nearly flat in group size.

The evaluation covers decode routing captured from three open MoE families on GSM8K and WildChat at group sizes from 8 to 64 GPUs, on torus placements with two replica slots per rank; the on-die engine's area and frequency come from the OpenROAD flow on the ASAP7 7 nm educational PDK, and the FPGA deployment is placed and routed to timing closure on the XCVH1582.

Absolute power is not reported because vectorless activity estimation diverges on this netlist's reconvergent arithmetic; the on-die engine's area and frequency come from global-route estimated parasitics on an educational PDK rather than silicon measurement; and the evaluation charges every decision as fully blocking, with no overlap with the layer's other work.