FedFit cuts federated LoRA fine-tuning communication by up to 100x while matching standard federated LoRA perplexity via dual vector-bank parameterization and quantization
FedFit targets the large communication overhead of federated fine-tuning of large language models and the LoRA aggregation dilemma between accurate Sum-of-Products (SoP) and communication-efficient Product-of-Sums (PoS); it reconstructs high-dimensional adapter matrices from two compact, disjoint global vector banks, alternates between decoupled single-bank updates and joint updates corrected by a Residual Spectral Aggregation mechanism, and adds blockwise quantization with client-side error feedback, achieving perplexity comparable to standard federated LoRA on Qwen2.5 models with compression ratios up to 100x higher, alongside theoretical convergence guarantees.