Voice-agent testing moves off manual dialing: LiveKit's Agent Simulations runs whole tasks with LLM users and fails the build
Synopsis
LiveKit moves voice-agent testing from turn-by-turn dialogue to task-level simulation: an LLM-driven simulated user plays out a scenario with a defined persona, goals and behavior, and an LLM judge grades the transcript against freeform expectations with a pass or fail verdict and a reason. Scenarios run in parallel, and results, transcripts and traces land in the LiveKit Cloud dashboard; text mode is the default, paced into spare capacity at low priority, and a failed scenario returns a non-zero exit code from the CLI. Serve AI used simulations to validate a full agent rewrite against a hundred real production calls before cutting any traffic over.
Interpretation
The unit of voice-agent testing becomes a whole task rather than a single exchange: the tester states what the user wants to accomplish, and the agent takes whatever path it needs to get there. The LiveKit Agents framework already supported turn-by-turn testing, but turn-by-turn testing cannot replicate the unpredictability of a real conversation, such as a caller who must give basic information before an appointment can be rescheduled, in an order that varies. Each written scenario stays in the suite, so when a tenth feature is added the checks for the first three run automatically and reveal whether the new work broke the old.
An LLM-driven user plays a scenario from start to finish, and an LLM judge then grades the transcript against freeform expectations. Each scenario can define the simulated user's persona, goals and behavior; once the conversation reaches a natural conclusion, the judge assigns a pass or fail verdict and the reason behind it. Scenarios run in parallel and results, transcripts and traces land in the LiveKit Cloud dashboard; Serve AI used simulations to validate a full agent rewrite against a hundred real production calls before cutting any traffic over.
Scenarios need not be written from scratch: a command-line tool reads the agent's code and generates scenarios grounded in its actual prompts, tools and conversation logic. Generation is grounded in the agent's own prompts, tools and conversation logic rather than a generic set of conversations. Generated scenarios are a starting point: keep the ones that surface something, adjust the ones that do not, and write your own for known cases; the checked-in scenarios.yaml becomes the source of truth, versioned alongside the agent it tests.
Perspective
The demonstrated setting is voice-agent testing inside LiveKit Cloud projects: text mode bypasses the speech models and tests only conversation logic, while audio mode covers the entire audio pipeline including background noise, interruptions and backchanneling, for cascaded or speech-to-speech full-duplex agents. Simulations run the real agent, so it calls its LLM as usual and in audio mode its STT and TTS too, billed at normal rates; getting started requires the LiveKit CLI, LiveKit Observability enabled and a LiveKit Cloud project.
Generated scenarios are explicitly a starting point, not a finished suite, and need human selection, adjustment and additions. Text mode has no audio and no pauses and runs strictly one turn at a time; in audio mode the simulated user can overlap the agent through a backchannel or by replying during a long pause. Agent Simulations is now generally available for all LiveKit Cloud projects and will work in concert with Agent Evaluations, coming soon after.