David Bensaïd

Low-rank adaptation (LoRA) writes the weight update as a product of two small matrices, but their norms often differ sharply during training, producing imbalanced gradient flow and unstable optimization. SingLoRA instead writes the update as a single low-rank matrix times its own transpose, removing inter-matrix scale conflicts by construction and roughly halving the trainable parameters. Fine-tuning LLaMA-7B on MNLI, SingLoRA reaches 91.3% accuracy against 89.1% for LoRA and 90.2% for LoRA+, while using only 60% of their parameter budget.

Low-rank adaptation (LoRA) writes the weight update as a product of two small matrices, but their norms often differ sharply during training, producing imbalanced gradient flow and unstable optimization. SingLoRA instead writes the update as a single low-rank matrix times its own transpose, removing inter-matrix scale conflicts by construction and roughly halving the trainable parameters. Fine-tuning LLaMA-7B on MNLI, SingLoRA reaches 91.3% accuracy against 89.1% for LoRA and 90.2% for LoRA+, while using only 60% of their parameter budget.