Expert pruning stops scoring experts one by one: HOPE counts their partnerships, and at 50% pruning beats the unpruned model on agentic coding
Synopsis
Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.
Interpretation
Expert pruning moves from giving each expert a single importance score to accounting for pairwise cooperation between experts, preserving cooperating groups when the pruning rate is high. Earlier pruning methods assume each expert's contribution is independent of all others and cut the lowest scalar scores within each layer, ignoring the stable cooperative groups that experts form when co-activated. Across 3 models, 2 calibration datasets and pruning rates from 10% to 50%, HOPE achieves an average rank of 2.07 among 5 methods, with top-1 finishes in 39% of conditions and top-2 finishes in 67%.
HOPE writes the change in layer output caused by pruning as a tractable surrogate whose expansion naturally produces cross-terms between pairs of experts. Minimizing the pruning error directly is intractable, because the error depends on which next-highest-gated experts replace the deleted ones, and those replacements depend on the prune-set itself. HOPE builds an interaction matrix F from gate weights and expert output norms on the calibration set, where diagonal entries measure individual importance and off-diagonal entries measure pairwise cooperative contribution, then solves a quadratic program via continuous relaxation in 1 to 2 seconds per layer.
The gap between HOPE and the first-order method REAP sits in the off-diagonal interaction terms, and HOPE gives up some individually high-scoring experts to keep tightly cooperating ones. The diagonal of F recovers REAP's expert rankings almost perfectly, with a Spearman correlation of 0.988, so the performance difference comes from the pairwise interaction terms. Among experts where HOPE and REAP disagree, experts retained only by HOPE consistently have higher average cooperation scores with the other retained experts.
Perspective
The demonstrated scope is three models, Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, calibrated on Evol-CodeAlpaca and SWE-Bench Verified trajectories, evaluated on the Tulu3 suite and SWE-Bench Pro, with pruning rates from 10% to 50% and each layer pruned independently.
At low pruning rates of 10% to 20%, first-order methods such as REAP and MAN are sometimes competitive and HOPE's advantage shrinks; HOPE prunes each layer independently and does not account for cross-layer interactions, and although a theoretical framework for cross-layer HOPE is presented, those results remain preliminary; the method captures pairwise interactions only, and higher-order interactions among triplets and beyond could add signal but are combinatorially difficult to estimate.