LoRA ranks no longer need to match: per-domain ranks cost averaging and utility selection nothing, while conflict-resolution merging loses up to 4 points
Synopsis
Model merging usually trains every expert at the same LoRA rank, yet each domain has its own best rank. Across 36 experts on mathematics, instruction following and multilingual data at four ranks and three model scales, six merge methods were compared at uniform ranks and at each domain's own best rank: simple averaging and loss-utility component selection are unaffected and gain up to 1.8 pp of multi-domain accuracy, while magnitude-based conflict-resolution methods such as TIES and KnOTS are inconsistent and can lose up to 4 pp. The cause is that selecting components by size discards the instruction adapter, whose singular values are smallest, almost entirely, and IFEval falls by 16 to 25 pp.
Interpretation
Experts do not need to share one LoRA rank: as long as the merge operator does not choose components by size, each domain can take the rank that suits it. Merging practice had trained every expert at the same rank, coupling the rank choice across experts even though each domain's best rank differs. On Qwen3.5 at 0.8B, 2B and 4B, moving each method from its best uniform-rank assignment to its best diverse one leaves simple averaging, SVD truncation and utility selection within the 1.4 pp resolution floor, with the largest single gain 1.8 pp for simple averaging at 0.8B.
The loss lands on instruction following rather than spreading evenly across the three domains. Magnitude truncation leaves the retained subspace badly unbalanced, with one expert holding at least 0.64 of the retained energy and another driven near zero in six of seven configurations. The instruction adapter has the smallest singular values, so it is dropped almost entirely at uniform rank and IFEval falls by 16 to 25 pp; selecting components by loss utility keeps all three experts and leaves accuracy nearly flat across an 8-fold rank range (0.7 pp, against 6.7 pp for TIES at 4B).
Which spectral components survive matters more to the merged result than how much energy they carry. Under uniform ranks about 65 to 84 percent of pooled components carry positive loss utility while the truncation budget admits roughly one third, and the count of useful components magnitude selection keeps out grows with rank. Utility selection retains less spectral energy yet merges better; at 4B the average-score gaps among GGSM, SVD, simple averaging and greedy soup fall inside the resolution floor and are treated as statistically tied.
Perspective
The results rest on the Qwen3.5 0.8B, 2B and 4B dense checkpoints, with all experts fine-tuned by QLoRA on a 4-bit quantized base, three task domains (mathematics, instruction following, multilingual) and a single seed per merge configuration. The authors propose more seeds and larger evaluations to confirm the small gains from diverse ranks.
The authors note that the first-order approximation can overestimate utility for some components, particularly those with extreme predicted scores, suggesting room for higher-order corrections; the small gains from diverse ranks sit below the 1.4 pp resolution floor and need more seeds and larger evaluations; the study covers three domains, LoRA fine-tuning only and one model family.