Model merging stops forcing every task into one subspace: GRAM groups by geometry and beats global merging at 83.8% on GLUE
Synopsis
Multi-task deployment often requires merging independently fine-tuned models into one, but existing merging methods compress all tasks into a single shared low-rank space, so tasks whose subspaces are misaligned lose task-relevant directions. GRAM uses only the fine-tuned weights, first partitioning tasks by the principal-angle geometry of their output subspaces and then running a standard SVD-based merge inside each group. On GLUE with Flan-T5-Base it reaches 83.8% average accuracy, above the 81.9% of global merging and the 81.7% of the strongest baseline TSV-Merge, at the cost of one extra deployed model.
Interpretation
Merging moves from one shared low-rank basis for all tasks to grouping by subspace geometry and merging separately within each group, so geometrically incompatible tasks no longer share one set of directions. Earlier merging methods apply a uniform global merge to all tasks, implicitly assuming sufficient subspace alignment; that assumption fails when tasks come from diverse domains, and tasks far from the bulk degrade most. On GLUE with Flan-T5-Base, GRAM averages 83.8% against 81.9% for global merging and 81.7% for the strongest baseline TSV-Merge; on the 8-task CLIP vision benchmark it averages 88.7% on ViT-B/32 and 94.0% on ViT-L/14, above Iso-CTS at 85.8% and 93.9%.
Grouping uses the angles between the output subspaces spanned by the left singular vectors of each task update; larger angles mean more task directions are discarded when one basis is shared. The pipeline computes the squared chordal distance between every task pair, averaged across layers, then clusters tasks with k-means++ on the Grassmannian, with the number of groups set by the deployment budget. On Flan-T5-Base with two groups, the ratio of intra-group to inter-group subspace similarity is 4.06 versus 0.87 for random grouping; agglomerative average-linkage clustering on the same geometry reproduces the identical partition, and the partition is stable across 10 initialization seeds.
Inside each group, rank budget is allocated adaptively by singular value, giving rank to the directions that most reduce reconstruction error rather than splitting it evenly across tasks. Uniform allocation wastes capacity on tasks with compact spectra and truncates long-tail tasks; the adaptive allocator pools all candidate singular values in a group and keeps the top entries. Ablation shows removing adaptive allocation lowers the GLUE average from 83.8% and the ViT-B/32 average from 88.8%; the module matters less in fully fine-tuned vision, which the authors attribute to more uniform full-fine-tuning spectra.
Perspective
The demonstrated scope is task-aware deployment where task identity is known at inference: Flan-T5-Base/Large on GLUE, CLIP ViT-B/32, ViT-L/14 and ViT-B/16 on the 8-task vision benchmark, and a case study with four public medical LoRAs on Qwen2-7B-Instruct; merging uses only fine-tuned weights, with no source data or calibration set.
Results are reported from a single seed, and the Flan-T5-Large margin is small and should be read with caution; each modality covers one model family, and the medical evaluation is a case study on public benchmarks rather than a real multi-site federated experiment; the theory analyzes idealized objects, an optimal basis and the Grassmannian k-means objective, rather than GRAM's actual column-stacked merge with adaptive allocation, so it should be read as geometric motivation rather than an end-to-end optimality guarantee.