Affective Adaptation in Multimodal Foundation Models: FFN Tuning Beats Attention Modules and Gate Projections Specialize Under Joint Optimization

Affective adaptation shows a consistent yet non-exclusive functional organization at the module level: under matched trainable-parameter budgets, adapting only the feed-forward network (FFN) consistently outperforms adapting only attention modules across all evaluated settings and, on average, nearly matches tuning all major Transformer projections. Prior understanding of how affective capabilities emerge within the internal architectures of multimodal affective foundation models was limited; this work provides a module-level analysis across 13 affective model instances, nine model designs, multiple scales, tasks, and training paradigms, identifying the FFN as a particularly efficient adaptation substrate. Evidence comes from module-level comparisons across 13 affective model instances and nine model designs under matched trainable-parameter budgets, covering multiple scales, tasks, and training paradigms.

The gate, up, and down projections exhibit comparable standalone adaptation capacity, yet their learned functional contributions become differentiated after joint optimization; module recovery and targeted interventions identify gate_proj as a particularly prominent pathway, and checkpoint analysis shows this differentiation develops over the course of training. The work characterizes this as emergent functional specialization: distinct pathway-level roles are not fully explained by standalone adaptation capacity but arise through joint affective adaptation, offering evidence on whether affective fine-tuning induces diffuse changes or organizes computation into functionally specialized pathways. Evidence comes from controlled functional analyses including module recovery, targeted interventions, and checkpoint analysis on representative models, showing differentiation develops over training.

Building on this finding, Gate-Focused Efficient Tuning (GET) retains 96.2-98.0% of the performance obtained by tuning all major Transformer projections while using only 19.3-24.5% as many trainable parameters. The result translates mechanistic insight about functional differentiation into a parameter-efficient adaptation strategy, showing that focusing on the gate pathway can substantially reduce trainable parameters while maintaining performance close to full-projection tuning. Evidence is a comparison between GET and tuning all major Transformer projections in terms of performance and trainable parameters, reporting 96.2-98.0% performance retention at 19.3-24.5% parameter share.

The work targets the adaptation mechanisms and parameter-efficient tuning of multimodal affective foundation models: its conclusions apply to the 13 affective model instances, nine model designs, and corresponding scales, tasks, and training paradigms analyzed, with controlled functional analyses validating them on representative models. For researchers and engineers seeking to reduce trainable parameters in affective adaptation or to understand the roles of the FFN and gate pathway, GET offers a directly referenceable adaptation strategy.

The loaded text is abstract-level content and does not include specific datasets, evaluation metrics, model scale details, or statistical tests, so the robustness of conclusions across tasks and scales cannot be judged; the timing of emergent functional specialization during training and the mechanistic explanation for gate_proj's prominence remain to be developed; how GET performs beyond affective tasks or other modality combinations is an open question.