# Model Merging

Multi-task deployment often requires merging independently fine-tuned models into one, but existing merging methods compress all tasks into a single shared low-rank space, so tasks whose subspaces are misaligned lose task-relevant directions. GRAM uses only the fine-tuned weights, first partitioning tasks by the principal-angle geometry of their output subspaces and then running a standard SVD-based merge inside each group. On GLUE with Flan-T5-Base it reaches 83.8% average accuracy, above the 81.9% of global merging and the 81.7% of the strongest baseline TSV-Merge, at the cost of one extra deployed model.

Multi-task deployment often requires merging independently fine-tuned models into one, but existing merging methods compress all tasks into a single shared low-rank space, so tasks whose subspaces are misaligned lose task-relevant directions. GRAM uses only the fine-tuned weights, first partitioning tasks by the principal-angle geometry of their output subspaces and then running a standard SVD-based merge inside each group. On GLUE with Flan-T5-Base it reaches 83.8% average accuracy, above the 81.9% of global merging and the 81.7% of the strongest baseline TSV-Merge, at the cost of one extra deployed model.

The work introduces a routing analysis toolkit and, across DeepSeekMoE, OLMoE and Qwen3-MoE, crosses source and merged router inputs and parameters and runs paired token- and task-level evaluations, finding that most expert reassignments are input-shift-induced rather than parameter-induced, that source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and that different expert selections can produce directionally similar mixture outputs; it therefore redefines routing failure as task loss recoverable under a specified routing intervention with non-routing parameters fixed, and proposes Selective Router Repair as a case study that finds no reliable evidence that source-specialist token-likelihood advantages identify beneficial local corr

The work introduces a routing analysis toolkit and, across DeepSeekMoE, OLMoE and Qwen3-MoE, crosses source and merged router inputs and parameters and runs paired token- and task-level evaluations, finding that most expert reassignments are input-shift-induced rather than parameter-induced, that source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and that different expert selections can produce directionally similar mixture outputs; it therefore redefines routing failure as task loss recoverable under a specified routing intervention with non-routing parameters fixed, and proposes Selective Router Repair as a case study that finds no reliable evidence that source-specialist token-likelihood advantages identify beneficial local corr

The work builds ACLArena, a four-stage sequential post-training pipeline spanning math, search, e-commerce, and instruction following; diagnoses forgetting and transfer at both the model level and the token level; systematically compares multi-teacher on-policy distillation (MMOPD), self-distilled fine-tuning (SDFT), and model merging; and proposes Mixture of Low-Rank Experts (MLE), which first consolidates multi-domain trajectories into a shared backbone via SDFT, then freezes the backbone and trains one RL-optimized LoRA expert per stage routed by environment context, improving both in-domain and out-of-domain performance on four reasoning and agentic tasks.

The work builds ACLArena, a four-stage sequential post-training pipeline spanning math, search, e-commerce, and instruction following; diagnoses forgetting and transfer at both the model level and the token level; systematically compares multi-teacher on-policy distillation (MMOPD), self-distilled fine-tuning (SDFT), and model merging; and proposes Mixture of Low-Rank Experts (MLE), which first consolidates multi-domain trajectories into a shared backbone via SDFT, then freezes the backbone and trains one RL-optimized LoRA expert per stage routed by environment context, improving both in-domain and out-of-domain performance on four reasoning and agentic tasks.