Different LLMs can now collaborate federatedly: FedVP pools locally trained verifiers instead of merging parameters
Synopsis
Federated learning usually requires institutions to share one model backbone so parameter updates can be aggregated, which keeps centers running different model families or sizes from collaborating. FedVP moves collaboration from parameter aggregation to shared verification: each client trains a verifier on its local data and candidate responses from its own frozen LLM, and the server pools these independently trained verifiers to score and select answers, never merging backbone parameters. Across five medical question-answering datasets, with centers using Qwen and LLaMA backbones from 3B to 14B parameters, one verifier pool is reused for all five backbones without retraining and raises mean accuracy by 1.46 to 5.79 percentage points over the strongest conventional federated baselines.
Interpretation
Federated collaboration no longer requires every institution to run the same LLM backbone, so centers with different model families and sizes can draw on each other's locally learned knowledge. Conventional federated fine-tuning relies on a shared backbone, where clients exchange compatible parameter updates for server-side aggregation, so differing architectures or scales block collaboration. Across five medical QA centers using LLaMA-3.2-3B, Qwen2.5-7B, LLaMA-3-8B, Qwen2.5-3B and Qwen2.5-14B backbones, FedVP improves every one of the five clients and beats the strongest conventional federated baseline by 1.46 to 5.79 percentage points in mean accuracy.
The carrier of collaboration is a pool of verifiers that judge whether a candidate response is correct without reading the parameters of the model that produced it. Each client fine-tunes a verifier with LoRA on local data and candidate responses sampled from its own frozen LLM, labeling each candidate by whether its extracted answer matches the reference, and updates only the adapters and scoring components. Trained verifiers enter the pool without parameter aggregation, each keeping its own backbone and scoring architecture; at inference every verifier scores the candidates, the scores are averaged, and the top-scoring response is selected, with the same pool reused for different target backbones without retraining.
Perspective
Evaluation is confined to five medical question-answering datasets, with centers using LLaMA and Qwen backbones from 3B to 14B parameters and following the FedCoT data organization; behavior outside this scope is not shown.
Evaluation covers only medical QA and the LLaMA and Qwen families, and the authors note that further evaluation across additional domains and a wider range of model families and sizes would help assess broader applicability; the distilled single verifier performs comparably to the teacher pool, and the authors suggest the small gains may reflect the extra fine-tuning stage that distillation introduces.