# Quadratic Program

Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

Mixture-of-experts models must hold every expert in memory at inference even though each token uses only a few, so deleting the least important experts is a common compression route. HOPE writes the pruning error as a sum of pairwise expert interaction terms, builds an interaction matrix from calibration data, and solves a quadratic program to choose the prune-set. Across Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air, HOPE holds an average rank of 1.58 at 50% pruning versus 2.42 for the next-best method, REAP; Qwen3.5-35B-A3B scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while REAP falls to 36.2.

Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.

Mixture-of-experts models must keep every expert in memory at inference even though each token uses only a few, so deleting experts by independent importance scores is a common fix. HOPE writes the pruning error as pairwise interaction terms between experts and selects a whole prune-set with a quadratic program, reaching the best average rank across three models, two calibration datasets and several benchmarks. On Qwen3.5-35B-A3B at 50% pruning it scores 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 while the first-order method REAP falls to 36.2.

The work introduces Priority-Constrained Descent (PCD), a gradient-based framework that anchors on the primary gradient and applies only the minimal Euclidean distortion needed to guarantee each secondary objective a τ-fraction of first-order progress, reporting better per-objective performance and Pareto dominance over symmetric multi-objective methods in structured pruning, unstructured sparsity with low-rankness, and synthetic experiments, with closed-form solutions for two and three objectives and scale invariance.

The work introduces Priority-Constrained Descent (PCD), a gradient-based framework that anchors on the primary gradient and applies only the minimal Euclidean distortion needed to guarantee each secondary objective a τ-fraction of first-order progress, reporting better per-objective performance and Pareto dominance over symmetric multi-objective methods in structured pruning, unstructured sparsity with low-rankness, and synthetic experiments, with closed-form solutions for two and three objectives and scale invariance.