Collaboration can backfire when modalities vanish: Lluna keeps a direct per-model prediction and gains 4.26% on average across eight datasets
Synopsis
Collaboration among pretrained unimodal models can become less accurate than a surviving unimodal model when modalities are missing. Lluna keeps each model's direct prediction as a private path, adds a graph-based common branch as a residual correction, and lets richer availability states guide poorer ones along single-modality removal chains. Across eight datasets in affective computing, scene understanding and medical prediction, it improves Mean-Missing performance by 4.26% on average over the strongest evaluated baseline, with positive collaboration gains at minimal inputs on all eight tasks.
Interpretation
As modalities disappear one by one, the gain from collaborating several pretrained unimodal models shrinks and can turn negative, so a surviving unimodal model whose own input is unchanged can be more accurate. Most missing-modality methods learn a single multimodal predictor using modality dropout, aggregation of available representations, or shared-specific modeling; whether collaboration itself stays beneficial under missing inputs was not tested separately. Across affective computing, scene understanding and medical prediction, fixing a surviving model as an anchor shows collaboration gains declining as modalities are removed and turning negative on some tasks.
Lluna makes common collaboration a correction rather than a replacement for unimodal evidence: each model passes through an adapter to a direct prediction, while a common branch passes messages on a compatibility graph weighted by prediction agreement and outputs a residual with a learned scale. Feature probes show modality-specific information is less recoverable after collaboration, which motivates designing the common branch as a residual correction to the private prediction instead of a substitute. Training applies Progressive Collaboration Consistency along single-modality removal chains, aligning the normalized common representation of a poorer state to its richer neighbor while task supervision adapts the full prediction to the available evidence.
On all eight datasets, Lluna beats the strongest evaluated baseline on Mean-Missing performance. The comparison spans general collaboration methods, missing-modality methods and Omni models, all sharing the same frozen model pool, pre-extracted representations, data splits and modality-availability masks as Lluna. Mean-Missing improves by 4.26% on average in relative terms; WorldSense accuracy rises from 30.73% to 33.90% and SIMS MAE falls from 35.90 to 34.24.
In the fixed-anchor collaboration diagnostic, Lluna keeps positive collaboration gains at minimal inputs on all eight tasks. In the same diagnostic, the equally weighted mean of seven baselines declines from full to minimal inputs on all eight tasks and is negative at minimal inputs on six of them. The diagnostic fixes one surviving unimodal anchor per modality and compares collaboration accuracy against that anchor across states retaining the modality.
Perspective
The evaluation covers eight datasets in affective computing, scene understanding and medical prediction, with missingness constructed as controlled removal of entire modalities under known availability states. The collaboration module is trained for a specified pool of frozen models, and pool-size and replacement experiments compare alternative pools after retraining the collaboration modules.
Evaluation centers on controlled removal of entire modalities; partial corruption, temporal misalignment and naturally occurring missingness tied to sample characteristics still need separate assessment. Changing the model pool requires retraining the collaboration module, and supporting pool changes without retraining the full module remains a direction for future work. Collaboration gains are dataset-level averages, and individual anchor/view comparisons can still be negative.