Skip to main content
Search

# Expert Pruning

4 related articles

Subscribe to updates
  1. Hugging Face Blog

    Expert pruning stops scoring experts one by one: HOPE counts their partnerships, and at 50% pruning beats the unpruned model on agentic coding

    Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

    HOPE (Higher-Order Pruning of Experts)Expert PruningMixture-of-ExpertsQuadratic ProgramAgentic coding
  2. Hugging Face - Blog

    Expert pruning stops scoring experts one by one: HOPE counts their partnerships and, at 50% pruning, beats the unpruned model on agentic coding

    Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.

    HOPE (Higher-Order Pruning of Experts)Expert PruningMixture-of-ExpertsQuadratic ProgramAgentic coding
  3. arXiv

    DEP prunes experts per request from prompts in multi-agent systems, beating static pruning and merging baselines most clearly when few experts are kept

    The work proposes Dynamic Expert Pruning (DEP): its analysis shows that in multi-agent systems different tasks and roles recruit different experts while static pruning applies one fixed mask to every request; DEP uses an agent's system and task prompts to produce a per-request expert mask in a single forward pass with no per-configuration calibration, achieving better overall accuracy than static pruning and merging baselines across diverse tasks, roles, model scales, and MoE architectures, generalizing to workflows unseen in training, with the largest margin when few experts are retained.

    Pohang University of Science and Technology
    Jabin Koo（First author）
    Expert PruningMixture-of-Expertsmulti-agent systems
  4. arXiv

    Higher-Order Pruning of Experts in Mixture-of-Experts Language Models

    The work introduces HOPE, a second-order expert-pruning objective that records a per-layer pairwise expert interaction matrix F during calibration and solves a quadratic program to select the prune-set, provably showing that first-order methods such as REAP are special cases that ignore interaction terms; across three frontier MoE models (up to 122B parameters), two calibration sets, and multiple benchmarks, HOPE achieves the best average rank in most conditions, with an average rank of 1.58 at 50% pruning (versus 2.42 for the next-best method, REAP) and gains of up to +6.1% on agentic coding.

    Alex M. Tseng（First author）
    HOPE (Higher-Order Pruning of Experts)Expert PruningMixture-of-ExpertsQuadratic ProgramAgentic coding

You’re all caught up