# Controllable Text-to-Speech

Audible's technology team applies controllable text-to-speech to audiobooks: a large language model first builds a fixed voice profile for each character and writes natural-language stage directions such as "whispering nervously" for each line, and a speech model then synthesizes from them. Character attribution reached 97% and 94% accuracy on 25 English and Spanish mystery, sci-fi and romance books; in a MUSHRA listening test, dialogue differentiation improved by 3.7 points, and eight of ten voices showed statistically significant listening gains across English and Spanish tests.

Audible's technology team applies controllable text-to-speech to audiobooks: a large language model first builds a fixed voice profile for each character and writes natural-language stage directions such as "whispering nervously" for each line, and a speech model then synthesizes from them. Character attribution reached 97% and 94% accuracy on 25 English and Spanish mystery, sci-fi and romance books; in a MUSHRA listening test, dialogue differentiation improved by 3.7 points, and eight of ten voices showed statistically significant listening gains across English and Spanish tests.

The work first shows that speech-text alignment only weakly predicts downstream acoustic similarity, then proposes Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner with GRPO using the teacher-forced likelihood of target speech tokens under a frozen TTS model as reward; on an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP outperforms the Base LLM and target-audio-informed captioning baselines on five downstream acoustic metrics and gains in LLM-judged contextual appropriateness and reference consistency.

The work first shows that speech-text alignment only weakly predicts downstream acoustic similarity, then proposes Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner with GRPO using the teacher-forced likelihood of target speech tokens under a frozen TTS model as reward; on an English subset of the ISCSLP 2026 CoT-TTS corpus, SRSP outperforms the Base LLM and target-audio-informed captioning baselines on five downstream acoustic metrics and gains in LLM-judged contextual appropriateness and reference consistency.