Expert pruning stops scoring experts one by one: HOPE counts their partnerships and, at 50% pruning, beats the unpruned model on agentic coding
Synopsis
Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.
Interpretation
Most of a mixture-of-experts language model's memory goes to keeping all experts resident at inference while each token activates only a few, and HOPE turns the choice of which experts to delete from per-expert scoring into a joint solve over expert partnerships. Earlier expert-pruning methods assign each expert a scalar importance score and cut the lowest-scoring experts layer by layer, assuming each expert's contribution is independent of the others. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, two calibration sets (Evol-CodeAlpaca and SWE-Bench Verified trajectories) and pruning rates from 10% to 50%, HOPE achieves an average rank of 2.07 among five methods against four first-order baselines, finishing first in 39% of conditions.
HOPE constrains the change in layer output caused by pruning with a computable surrogate whose expansion naturally produces cross terms between pairs of experts. The surrogate strictly upper-bounds the dominant component of the pruning error, and its off-diagonal entries measure the average gate-weighted output product of two experts over tokens where both are co-activated. The interaction matrix for GLM-4.5-Air shows bright blocks along the diagonal, indicating cooperative clusters that depend on layer; its diagonal recovers REAP's expert rankings almost perfectly (Spearman ρ = 0.988), so gains over REAP come from the off-diagonal terms.
The selection is written as a 0/1 vector, and minimizing the surrogate's expectation over the calibration set becomes a quadratic program. Minimizing the pruning error directly is intractable because the error depends on the identities of replacement experts, which in turn depend on the prune-set itself. The quadratic program is solved by continuous relaxation and then rounded to a binary prune-set, taking 1 to 2 seconds per layer, with total overhead negligible next to the calibration forward pass.
Perspective
The demonstrated scope is three models (Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air), calibration on Evol-CodeAlpaca and SWE-Bench Verified trajectories, evaluation on the Tulu3 suite and SWE-Bench Pro, and pruning rates from 10% to 50%.
HOPE's advantage is most pronounced at high pruning rates of 40% to 50%; at 10% to 20%, first-order methods such as REAP and MAN are sometimes competitive and the advantage shrinks. HOPE prunes each layer independently and does not account for cross-layer interactions, and although a theoretical framework for cross-layer HOPE is presented, those results remain preliminary. The method captures pairwise interactions only; even higher-order interactions such as triplets could add signal but are combinatorially difficult to estimate.