Audiobooks stop reading in one voice: Audible fixes a voice profile per character and lifts dialogue differentiation by 3.7 points

Long-form audiobook narration now assigns a fixed voice per character instead of having one general speech model read the whole book in a single manner. General-purpose text-to-speech models target interactions lasting seconds, such as chatbots and virtual assistants, and on audiobooks they fail to reliably change speaking style between characters as a human narrator would, so listeners struggle to separate narration from dialogue and to tell which character speaks which lines. On 25 English and Spanish books in the mystery, sci-fi and romance genres, with ground truth from two human annotators, BookLLM identified which character speaks where, including separating narration from dialogue, at 97% accuracy in English and 94% in Spanish.

At synthesis time each unit of text is passed to the speech model together with its speaker's voice profile, so a character's voice stays fixed across the whole book. The pipeline has two parts: BookLLM reads the entire book to build a cast list, ranks characters by importance and assigns each a distinct profile from a curated voice catalogue, deliberately maximizing contrast among prominent characters while minor and one-off characters fall back to the default narrator; it then attributes every passage to a speaker and writes a caption for each dialogue line. Those captions are exactly the natural-language speech descriptions the controllable-TTS model was trained to follow, such as "whispering nervously" or "shouting in anger"; in the standardized MUSHRA listening test, dialogue differentiation improved by 3.7 points, and eight of ten voices showed statistically significant listening improvement in the English and Spanish tests.

A controllable speech model needs more than text-speech pairs: it needs annotations describing how the speech sounds, and such data was scarce. Previous controllable TTS models could not handle highly expressive speech, so most existing training data is neutral in tone, and existing prosody-annotated datasets each use their own format, some long, some short, some single tags, which tends to make a model overfit specific formats. Training data was expanded through new recordings and human captioning of existing audio, with an LLM rewriting original prompts into different lengths, verbosity and structures to improve generalization; on the synthetic side, expert models and signal-processing algorithms such as pitch extraction, speaking rate extraction and emotion detection feed numerical features to an LLM that generates captions, while a speech captioner performs the reverse task, and the two are improved iteratively as a synthetic-data flywheel.

The demonstrated scope is audiobook synthesis: character and dialogue attribution evaluated on 25 English and Spanish mystery, sci-fi and romance books, and dialogue differentiation plus ten voices compared in English and Spanish MUSHRA listening tests.

Character attribution accuracy is reported per language, 97% in English and 94% in Spanish; listening improvement reached statistical significance for eight of ten voices, not all of them.