Alex M. Tseng

The work introduces HOPE, a second-order expert-pruning objective that records a per-layer pairwise expert interaction matrix F during calibration and solves a quadratic program to select the prune-set, provably showing that first-order methods such as REAP are special cases that ignore interaction terms; across three frontier MoE models (up to 122B parameters), two calibration sets, and multiple benchmarks, HOPE achieves the best average rank in most conditions, with an average rank of 1.58 at 50% pruning (versus 2.42 for the next-best method, REAP) and gains of up to +6.1% on agentic coding.

The work introduces HOPE, a second-order expert-pruning objective that records a per-layer pairwise expert interaction matrix F during calibration and solves a quadratic program to select the prune-set, provably showing that first-order methods such as REAP are special cases that ignore interaction terms; across three frontier MoE models (up to 122B parameters), two calibration sets, and multiple benchmarks, HOPE achieves the best average rank in most conditions, with an average rank of 1.58 at 50% pruning (versus 2.42 for the next-best method, REAP) and gains of up to +6.1% on agentic coding.