Low-rank adaptation learns one matrix: SingLoRA beats LoRA and LoRA+ on LLaMA-7B at 60% of their parameter budget
Synopsis
Low-rank adaptation (LoRA) writes the weight update as a product of two small matrices, but their norms often differ sharply during training, producing imbalanced gradient flow and unstable optimization. SingLoRA instead writes the update as a single low-rank matrix times its own transpose, removing inter-matrix scale conflicts by construction and roughly halving the trainable parameters. Fine-tuning LLaMA-7B on MNLI, SingLoRA reaches 91.3% accuracy against 89.1% for LoRA and 90.2% for LoRA+, while using only 60% of their parameter budget.
Interpretation
The low-rank update is no longer a product of two matrices but a single low-rank matrix times its own transpose, so a scale difference between two factors cannot arise. In LoRA and its variants the two factors often differ greatly in norm during training, requiring separate learning rates for the two matrices or a specialized optimizer to stay stable. Fine-tuning LLaMA-7B on MNLI, SingLoRA reaches 91.3% accuracy, LoRA 89.1% and LoRA+ 90.2%, while SingLoRA uses only 60% of their parameter budget.
Training stability comes from the parameterization itself, so standard first-order optimizers can be used directly without tuning two learning rates. Earlier fixes either set different learning rates for the two matrices or introduced dedicated optimizers such as Riemannian gradient descent, adding tuning work in practice. Fine-tuning LLaMA-7B on MNLI over learning rates from 5×10⁻⁵ to 10⁻⁴, SingLoRA's accuracy fluctuates by about 1%, while LoRA varies by up to 4.8%.
The symmetric parameterization does not weaken attention expressiveness, because the product of the symmetric query and key updates can still yield a non-symmetric attention transformation. A single matrix times its transpose looks more restricted than two free matrices, so it needs checking whether it covers the attention patterns LoRA can learn. On a 128×128 synthetic attention-fitting task averaged over 1,000 random initializations, SingLoRA reaches an approximation error of about 10⁻⁵ while LoRA plateaus near 10⁻².
Perspective
Experiments cover RoBERTa-base and GPT-2 on the GLUE tasks MNLI, QQP and QNLI, LLaMA-7B on MNLI, Qwen2.5-1.5B on GSM8K, and Stable Diffusion V1.5 fine-tuned on DreamBooth and on a 40-face dataset; all runs used a single NVIDIA A40 GPU with 48GB of memory.
On GSM8K at equal rank, SingLoRA reaches 61.0% with 37% fewer parameters, below LoRA's 64.5%; it exceeds LoRA's 64.5% only at a comparable parameter budget, with 66.5%. Image-generation DINO similarity is 0.151 for SingLoRA, 0.148 for DoRA and 0.143 for LoRA, a small margin. Combining with variants such as DoRA is listed as a future direction without experimental results.