Emotional TTS stops blending speaker style into emotion: HiP-ETTS splits prompts into shared and complementary factors and reaches 2.98% WER under audio prompts
Synopsis
Multimodal emotional text-to-speech previously packed cross-speaker emotion regularities and individual expression into one emotion condition, so strengthening emotion could damage linguistic content. HiP-ETTS splits prompt-derived emotion into an emotion-shared factor and a complementary emotion factor, anchors linguistic structure with a hierarchical emotional prior, and refines emotional detail through adaptive projection and flow matching. Across audio, text and facial-image prompts its word error rate is lower than or comparable to prior methods, and its VAD-CCC and Emo-SIM emotion-consistency scores are the highest in every prompt condition, reaching 0.760 and 0.768 under audio prompts.
Interpretation
Emotional speech synthesis can now raise emotional consistency while preserving linguistic content, with one model accepting speech, text or facial images as the emotion prompt. Earlier multimodal emotional prompting injected emotion as a single condition without explicitly separating cross-speaker emotional regularities from speaker-specific expressive variation, so the emotion condition could conflate personal expression with the target emotion. Under audio prompts HiP-ETTS reaches 2.98% word error rate, 0.760 VAD-CCC and 0.768 Emo-SIM, against 6.59%, 0.676 and 0.666 for MaskGCT and 9.27%, 0.641 and 0.611 for SparkTTS under the same prompt condition.
The model first splits the emotion prompt into two streams, then generates acoustics in two stages: a content-anchored emotional prior is built first, and emotional detail is filled in along a flow-matching trajectory. The emotion-shared factor is the spherical emotion representation from a VAD extractor, while the complementary factor is the residual of a high-dimensional emotion embedding that VAD does not explain; the two feed the shared stream and the speaker-conditioned specific stream of a two-stream prosody predictor. The hierarchical prior combines a content backbone with prosody, timbre and emotion residual offsets, the emotion residual entering through FiLM modulation; a shallow-state projection head then estimates an intermediate state and its flow time, constrained between 0.25 and 0.90, and the decoder models only the remaining trajectory to the endpoint.
The two emotion factors and the two-stage refinement carry distinct roles, and removing either mainly costs fine-grained emotion rather than linguistic content. Replacing the complementary emotion factor with one from another speaker of the same emotion category and similar VAD values changes word error rate and VAD-CCC only slightly, while Emo-SIM falls from 0.768 to 0.707. Replacing the two-stream prosody predictor with a single stream raises word error rate from 2.98% to 3.51% and drops VAD-CCC and Emo-SIM to 0.439 and 0.566; removing the emotion residual offset lowers Emo-SIM to 0.674 while word error rate is 3.11%.
Perspective
The evaluation covers zero-shot emotional speech synthesis under audio, text and facial-image emotion prompts, with baselines run on their officially released checkpoints and inference settings and efficiency metrics measured under one shared inference setting. The authors propose future work on robustness to noisy or adversarial multimodal cues.
Under text prompts HiP-ETTS reaches 2.99% word error rate, slightly above CosyVoice2's 2.56%, which the authors attribute to the latter's large-scale text-conditioned training; HiP-ETTS shows no overall UTMOS advantage, and the authors note that higher UTMOS among these baselines does not necessarily mean better content, emotion or prosody. Under facial prompts VAD-CCC is 0.439, well below the 0.760 seen with audio prompts.