FedFit targets the large communication overhead of federated fine-tuning of large language models and the LoRA aggregation dilemma between accurate Sum-of-Products (SoP) and communication-efficient Product-of-Sums (PoS); it reconstructs high-dimensional adapter matrices from two compact, disjoint global vector banks, alternates between decoupled single-bank updates and joint updates corrected by a Residual Spectral Aggregation mechanism, and adds blockwise quantization with client-side error feedback, achieving perplexity comparable to standard federated LoRA on Qwen2.5 models with compression ratios up to 100x higher, alongside theoretical convergence guarantees.
FedFit targets the large communication overhead of federated fine-tuning of large language models and the LoRA aggregation dilemma between accurate Sum-of-Products (SoP) and communication-efficient Product-of-Sums (PoS); it reconstructs high-dimensional adapter matrices from two compact, disjoint global vector banks, alternates between decoupled single-bank updates and joint updates corrected by a Residual Spectral Aggregation mechanism, and adds blockwise quantization with client-side error feedback, achieving perplexity comparable to standard federated LoRA on Qwen2.5 models with compression ratios up to 100x higher, alongside theoretical convergence guarantees.