# multimodal transformer

Using PTR-ToF-MS headspace volatile fingerprints plus HPLC- and amino-acid-analysis-quantified non-volatile metabolites, this study built a two-stream Transformer that encodes each modality separately and fuses them for four-class grading of Baimudan white tea, reaching 95.8% accuracy (23/24) and a 0.958 macro-F1 on an independent prediction set drawn from 120 samples (30 per grade; 96 training, 24 prediction), with a single Special-grade sample misclassified as Grade I, mean cross-validated accuracy of 0.979±0.026 within the training set, and SHAP/attention analyses linking high grades to floral/sweet volatile ions plus higher amino acids and soluble sugars and lower grades to greener/woody volatile ions and kaempferol-related markers.

Using PTR-ToF-MS headspace volatile fingerprints plus HPLC- and amino-acid-analysis-quantified non-volatile metabolites, this study built a two-stream Transformer that encodes each modality separately and fuses them for four-class grading of Baimudan white tea, reaching 95.8% accuracy (23/24) and a 0.958 macro-F1 on an independent prediction set drawn from 120 samples (30 per grade; 96 training, 24 prediction), with a single Special-grade sample misclassified as Grade I, mean cross-validated accuracy of 0.979±0.026 within the training set, and SHAP/attention analyses linking high grades to floral/sweet volatile ions plus higher amino acids and soluble sugars and lower grades to greener/woody volatile ions and kaempferol-related markers.

Ogata and Nakashima ask where encoding happens when multimodal language models skip dedicated perceptual encoders and expose a shared transformer to lightly projected patches, audio frames, or discrete visual tokens, and through linear probing, similarities to perceptual encoders, and causal analyses they find the transformer internalizes the missing computation, constructing task-usable perceptual representations within its own early-to-middle layers, a structure they call a Virtual Encoder.

Ogata and Nakashima ask where encoding happens when multimodal language models skip dedicated perceptual encoders and expose a shared transformer to lightly projected patches, audio frames, or discrete visual tokens, and through linear probing, similarities to perceptual encoders, and causal analyses they find the transformer internalizes the missing computation, constructing task-usable perceptual representations within its own early-to-middle layers, a structure they call a Virtual Encoder.