Discover Artificial IntelligenceSource publication:
Loading content…
Recognition of psychological stress in dialogue has mostly treated each utterance as an isolated fragment and concatenated modality features at the final layer, which cannot capture how emotion accumulates and shifts across turns. BERT-EmoShiftNet pairs BERT's multi-level semantic encoding with EmoShiftNet's emotion-transition modelling, aligns text, speech and vision through a cross-modal Transformer, and unifies them with cross-modal contrastive learning in a shared embedding space. On a dialogue-level split of a self-built dataset, full-modal five-class accuracy is 85.0% (95% CI [83.2%, 86.8%]), emotion-transition prediction accuracy is 76.2%, and mean detection latency is 13.4 s.
Stress recognition in dialogue moves from per-sentence judgement to cross-turn modelling: alongside the current stress category, the model predicts which way emotion shifts between adjacent turns. Earlier multimodal emotion-recognition frameworks mostly used late fusion, concatenating modality-specific features only at the final classification layer and treating emotion as a static per-sentence label, without modelling accumulation or transfer across dialogue turns. On a self-built dataset split strictly at the dialogue level (6391 training, 1484 validation, 2227 test samples), full-modal five-class accuracy is 85.0% with a 95% confidence interval of [83.2%, 86.8%]; emotion-transition prediction accuracy is 76.2% with a mean detection latency of 13.4 s.
Text, speech and vision are each encoded separately, then aligned in one representation space before the emotion-transition module tracks state changes. The BERT encoder captures fine-grained shifts in language semantics, a convolutional-recurrent network extracts speech time-frequency features, a convolutional neural network with optical flow obtains facial-expression dynamics, and a shared embedding space with cross-modal contrastive learning yields a unified representation. The cross-modal Transformer uses text features as queries and speech and visual features as keys and values, so attention is allocated by semantic context; mean correlation between speech prosody and text semantic features rises from 0.217 before alignment to 0.583 after dynamic time warping alignment, and removing the alignment module lowers overall test accuracy by 4.2 percentage points.
The emotion-transition matrix is learned end to end, and its structure is domain-general while its specific probabilities reflect the dynamics of this stress-dialogue dataset. The matrix is initialized from a uniform transition distribution and optimized jointly with all other network parameters by backpropagation through the classification loss, gradually fitting the statistical emotion-transition patterns in the training corpus. The highest-probability paths are positive stress to non-affective (0.312), stressful anger to stressful sadness (0.247) and fear of stress to stressful sadness (0.221); against a first-order Markov baseline (transition accuracy 0.684) the matrix improves accuracy by 7.8 percentage points, and in leave-one-subject-out cross-validation transition accuracy drops only 2.8 points versus the mixed-speaker setting.
The framework is positioned as an exploratory computational framework rather than a clinical diagnostic device, intended to assist clinicians by supplying quantitative behavioural indicators for further evaluation. The data comprise 10,102 dialogue turns from 236 participants, collected and annotated by the Psychology Laboratory of Changchun Medical College, with each turn independently labelled by three clinical psychologists and a Fleiss kappa of 0.78 across five categories. Results cover five-class stress-dialogue recognition, emotion-transition prediction and several robustness tests; on an independent subset from a different participant cohort, overall accuracy remains 82.3%.
The 97% peak accuracy is attributed inconsistently in the source: the abstract and conclusion call it a full-modal peak on the complete five-class stress-dialogue task, while the experimental analysis section calls it a simplified binary stress versus non-stress subtask under full-modal input; five-class full-modal accuracy is 85.0%. The source also gives two accounts of an earlier 99% figure, one calling it an unreported preliminary ablation and another a simplified binary auxiliary ablation, and that figure has been removed from the conclusion. The data are markedly imbalanced, with positive stress at 52.6% of the training set and fear of stress at only 6.6%, so per-class metrics and class-weighted and focal losses are reported. Emotion-transition accuracy accumulates with turns, starting at 62.1% in the first round and stabilizing at 76.2% after 5 to 6 turns.