# HOPE (Higher-Order Pruning of Experts)

Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.

Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.