Speech translation stops carrying only words: Spanish tone is projected into English expression space, then fed to a frozen generator

Speech-to-speech translation now has a separate control channel for expression: semantic content, speaker identity, and utterance-level expression are split into three channels that meet only at synthesis time. Existing end-to-end systems, speech MLLMs, and cascaded voice-cloning systems can preserve content or timbre, but they lack an explicit mechanism for how source expression should be realized in the target language. On the bidirectional Spanish–English TOPI benchmark, PACT leads on ASR-BLEU, pragmatic similarity, and target-referenced speaker similarity in both translation directions against the evaluated end-to-end, MLLM, and cascaded baselines.

The method first turns source pragmatic cues into a target-language expression vector, then turns that vector into continuous prompts a frozen generator can read. Injecting the source pragmatic vector directly is not appropriate, because the selected HuBERT dimensions are language-specific and the source vector is not aligned with the target-language expression space. The cross-lingual projector is fitted on paired pragmatic vectors from the DRAL training split using partial least squares; the target-language mean and target-side realization subspace are also estimated only from the training split, with no evaluation-split statistics used.

The projected vector passes through a conservative realization calibration before a lightweight adapter converts it into soft prompts. The projected vector may still contain directions the frozen generator cannot reliably realize, which would fall outside the target-language expression manifold. Calibration constrains the residual to a target-side partial least squares subspace, with subspace dimension and shrinkage coefficient selected on the development set and fixed before evaluation; it is enabled for Spanish-to-English and disabled for English-to-Spanish based on development-set performance.

The report covers evaluation on the bidirectional Spanish–English TOPI benchmark and a subjective study on 10 Spanish-to-English utterances rated by five proficient English speakers. The intended readers are engineers and researchers working on expressive speech translation, cross-lingual prosody, and pragmatic transfer; the method trains only the cross-lingual projector, realization calibration statistics, and a lightweight adapter, with the speech generator frozen throughout.

Realization calibration is disabled for English-to-Spanish because the Spanish-target adapter is trained only on DRAL without large-scale Spanish expressive pretraining, so calibration may over-constrain the target-space prompt representation. The subjective evaluation covers only the Spanish-to-English direction, and naturalness and expression preservation are not identical objectives: Qwen3.5-Omni-Plus receives the highest naturalness score but is substantially weaker in target-referenced speaker similarity and pragmatic delivery.