Federated fine-tuning stops choosing sides: shared adapters plus a local vision projection hit 82.15 harmonic mean across ten datasets

Federated vision-language fine-tuning can now keep both a client's local accuracy and its ability to recognize unseen classes, instead of trading one for the other. Most earlier federated vision-language work pursued personalization alone, and only a few studies aimed at personalization and generalization together. Across ten image datasets, BLEND reaches an average harmonic mean of 82.15, 4.81 points above the strongest baseline, with a cross-dataset standard deviation of 14.40 against that baseline's 16.46.

The recipe inserts trainable global adapters into both the vision and text encoders of CLIP, then adds a second branch at the vision projection head that belongs to one client only. The global adapters are trained locally and aggregated periodically at the server, while the personalized projection adapter is never uploaded and stays on the client. Training minimizes a weighted sum of a cross-entropy loss on the fused representation and a running anchor loss that keeps the global vision representation aligned with the text representation, with the best harmonic mean at an anchor weight of 0.2.

Which parameters get aggregated sets where the personalization-generalization balance lands. Aggregating every trainable parameter raises generalization but clearly hurts personalization, while aggregating nothing keeps personalization but drops generalization. On average, aggregating only the global vision and text adapters while keeping the personalized projection adapter local gives a harmonic mean of 82.15, above full aggregation at 79.01, no aggregation at 77.71, vision-only at 71.82, and text-only at 77.04.

The method targets CLIP-based image-text classification, trained on ten image datasets with 16 shots per class and Dirichlet-simulated non-IID client data, and validated on the real-world iNaturalist-User-120k distribution with larger client counts. It fits federated fine-tuning settings that keep a private adapter on each client while sharing part of the parameters. Extending it to audio, video, retrieval, generative tasks, and multi-task settings requires redesigning the fusion mechanism and the anchoring objective.

The method currently covers CLIP-style image-text modalities, and extension to other multimodal architectures, retrieval, and generative tasks remains open. Performance depends on hyperparameters that govern the personalization-generalization trade-off, and their adaptive selection in new environments is unsolved. Extension to fully decentralized, serverless federated vision-language models, and the integration of privacy-preserving aggregation such as differential privacy, are still open questions.