ElevenLabs brings 14 audio models to AI Gateway, letting one key cover transcription, live voice and music
Synopsis
Vercel and ElevenLabs are partnering to bring 14 ElevenLabs audio models to AI Gateway, spanning speech to text, streaming text to speech, realtime voice agents, music, sound effects and multi-voice dialogue. Previously these capabilities had to be wired up separately; now an app or agent can transcribe a user's speech, reason over it with a language model of its choice and answer in an ElevenLabs voice through a single gateway API key, with every request landing in the same usage, cost and trace views as other AI Gateway traffic. All ElevenLabs models are 50% off on AI Gateway for the first 14 days, through November 9th.
Interpretation
An app or AI agent can call ElevenLabs audio capabilities with a single AI Gateway key, chaining transcription, language-model reasoning and spoken reply in one interaction. Previously these audio capabilities required separate integrations, with transcription, the language model and speech synthesis configured independently. Every request lands in the same usage, cost and trace views as other AI Gateway traffic, showing the three kinds of model calls sit under one metering setup.
The six audio model types split the work: Scribe v2 Realtime turns speech into text while the user is still talking, and Eleven Flash v2.5 reads the language model's answer aloud as it streams in with low latency. Streaming text to speech lets the user hear the reply begin before the language model finishes writing it, rather than waiting for the full answer. The example code sends a 16 kHz microphone stream to Scribe v2 Realtime, passes the transcript to a language model, and pipes the text stream straight into Eleven Flash v2.5's audio stream.
Realtime voice agents run the whole spoken conversation inside one WebSocket session on Eleven Realtime v1 Max and v1 Mini, while the music and sound-effect models generate audio from a text description. The realtime models need no separate transcription or speech step, and multi-voice dialogue is produced in a single request by Eleven v4, v4 Turbo and v3. The browser opens the WebSocket session itself and the server only mints a short-lived token, keeping the API key off the client; the realtime models support semantic voice activity detection but not server-vad.
Perspective
The material shows the 14 ElevenLabs models callable on AI Gateway and how to call them, covering speech to text, streaming text to speech, realtime voice agents, music, sound effects and multi-voice dialogue, for developers adding audio to an app or agent.
The realtime voice models support semantic voice activity detection but not server-vad, a limit stated in the material; the material gives no latency, accuracy or cost figures for the models.