In federated learning with missing modalities, the server should count prediction support: RASO lifts KU-HAR from 41.30 to 49.13 under concatenation
Synopsis
In multimodal federated learning, when clients observe only some modalities, aggregating by which modality is available drops encoders that still shape predictions. RASO instead picks each component's contributors by prediction dependence and local-optimizer membership, keeps momentum on encoders, and synchronizes the shared prediction block to its current aggregate. Across KU-HAR, CREMA-D and other datasets under two fusion architectures, concatenation raises KU-HAR macro-F1 from 41.30 under the availability rule to 49.13, while under masked attention RASO leads on UCI-HAR, KU-HAR and CREMA-D and FedOpt leads on CrisisMMD.
Interpretation
When a federated server aggregates multimodal updates, it must separate which modalities a client observes from which components actually took part in its local learning, because an encoder for a missing modality can still receive task gradients through missing-input handling. Selecting contributors by modality availability skips the missing-modality encoder's update under concatenation while keeping the prediction-block updates learned alongside it. In concatenation diagnostics on KU-HAR and CREMA-D, the absent encoder's gradient share averages 0.206 and 0.148, and the cosine between its missing-input and complete-input gradients ranges from 0.027 to 0.183.
RASO determines contributors separately for each component, then applies a server update matched to that component's role. Earlier server optimization applied momentum to the client aggregate uniformly, without distinguishing encoders from the shared prediction block. With masked attention and the local-training protocol and contributor rule fixed, prediction-block momentum raises scores under current encoder aggregation but lowers them under persistent encoder updates on all three datasets.
Encoder updates accumulate residuals with momentum, while the shared prediction block is synchronized directly to its current aggregate rather than following a historical direction. Historical prediction-block momentum paired with persistent encoder updates performs worst, showing the two update policies must be chosen together. Under masked attention, current encoders with a current prediction block score 55.00, 39.23 and 42.46 on UCI-HAR, KU-HAR and CREMA-D, while persistent encoders with a persistent prediction block score 5.31, 32.47 and 42.27.
In the main 200-round comparison, RASO reaches the highest or near-highest macro-F1 on most datasets and fusion structures. Against FedAvg, FedOpt, FedProx, PmcmFL, MFCPL, FedMobile, BMSFed and BLOSSOM, RASO leads UCI-HAR, KU-HAR and CREMA-D under masked attention, and under concatenation leads CREMA-D and exceeds FedOpt on all four datasets. Under masked attention RASO scores 63.33, 48.30 and 48.52 on UCI-HAR, KU-HAR and CREMA-D, and 59.41, 49.13 and 48.20 under concatenation; on CrisisMMD FedOpt leads at 11.00 under masked attention and FedMobile leads at 11.28 under concatenation.
Perspective
Evaluation covers four datasets, UCI-HAR, KU-HAR, CREMA-D and CrisisMMD, with two-modality concatenation and masked-attention fusion, 10% participation, one local epoch, batch size 16 and terminal evaluation at round 200; contributor diagnostics use concatenation only, and support and availability coincide under masked attention.
The interaction where prediction-block momentum helps under current encoder aggregation but hurts under persistent encoder updates was observed under masked attention with a fixed local protocol; matched-step replay shows availability-based candidates have lower immediate DEV loss, opposite to the terminal comparison; and at matched training/test missing rates, skipping fully missing clients changes the population across rates.