# Federated LoRA Fine-Tuning

This work measures per-round upload and download bytes, rather than parameter-count ratios, for five federated LoRA protocols and places them on a single communication-quality frontier scored by held-out instruction-following loss; ReverseAdaptive, which learns both LoRA factors and then freezes one once the relative improvement in training loss falls below a dimensionless threshold, sits at the knee of that frontier, cutting measured round-trip communication by 40.5% relative to FLoRA at a held-out loss cost of 0.0063 and beating FFA-LoRA, which freezes that factor at initialization, by 0.0182 in held-out loss, more than twenty times the largest per-method seed standard deviation, with the same threshold carrying to LLaMA-3.2-3B without retuning for a 30.0% saving.

This work measures per-round upload and download bytes, rather than parameter-count ratios, for five federated LoRA protocols and places them on a single communication-quality frontier scored by held-out instruction-following loss; ReverseAdaptive, which learns both LoRA factors and then freezes one once the relative improvement in training loss falls below a dimensionless threshold, sits at the knee of that frontier, cutting measured round-trip communication by 40.5% relative to FLoRA at a held-out loss cost of 0.0063 and beating FFA-LoRA, which freezes that factor at initialization, by 0.0182 in held-out loss, more than twenty times the largest per-method seed standard deviation, with the same threshold carrying to LLaMA-3.2-3B without retuning for a 30.0% saving.

FedFit targets the large communication overhead of federated fine-tuning of large language models and the LoRA aggregation dilemma between accurate Sum-of-Products (SoP) and communication-efficient Product-of-Sums (PoS); it reconstructs high-dimensional adapter matrices from two compact, disjoint global vector banks, alternates between decoupled single-bank updates and joint updates corrected by a Residual Spectral Aggregation mechanism, and adds blockwise quantization with client-side error feedback, achieving perplexity comparable to standard federated LoRA on Qwen2.5 models with compression ratios up to 100x higher, alongside theoretical convergence guarantees.

FedFit targets the large communication overhead of federated fine-tuning of large language models and the LoRA aggregation dilemma between accurate Sum-of-Products (SoP) and communication-efficient Product-of-Sums (PoS); it reconstructs high-dimensional adapter matrices from two compact, disjoint global vector banks, alternates between decoupled single-bank updates and joint updates corrected by a Residual Spectral Aggregation mechanism, and adds blockwise quantization with client-side error feedback, achieving perplexity comparable to standard federated LoRA on Qwen2.5 models with compression ratios up to 100x higher, alongside theoretical convergence guarantees.