ElevenLabs unveils an interaction layer in New York: voice agents that can be fine-tuned, remember customers, and switch models by conversational turn

ElevenLabs frames the launch as building an interaction layer for AI, so that businesses reach customers and employees through voice and chat agents rather than only through speech models. The company started with voice and is now expanding to the models and tools behind all the ways people interact with technology. The announcements were made onstage at the New York Summit by CEO and co-founder Mati Staniszewski, who said conversational agents will become critical enterprise infrastructure within a few years, as essential as cloud or payments.

Domain-specific models are one thread: Scribe v3 Medical is a transcription model for healthcare that recognizes thousands of complex medical terms and distinguishes drug names such as cycloserine and cyclosporine. General transcription models do not separate industry vocabulary, while each industry has its own language and preferred channel. The company says the model recognizes thousands of complex medical terms with much greater accuracy, and it said a domain-optimized model for financial services is in the works.

In the coming months, ElevenLabs plans to help businesses fine-tune certain open-source language models and its own voice models using data from their live deployments, optimizing continuously against metrics the business chooses, such as customer satisfaction (CSAT) or reliability. Businesses previously used models pretrained for an industry but could not refine them for their own context and customers. Staniszewski compared the process to each company having its own reinforcement-learning stack, with businesses working alongside ElevenLabs researchers on the fine-tuning.

The previewed Interaction Engine starts from the rhythm of a conversation, handling turn-taking and interruptions so the agent knows when to jump in and when to stop, and it selects the right LLM for each part of an exchange. Not every agentic task needs a frontier model at every step, and deciding which models to use, how and when is itself a problem to solve. According to the description, a cheaper model can handle triage to keep token use in check, while a frontier model steps in for more complex tasks.

The report covers the products, previews and partnerships ElevenLabs announced at its New York Summit: the model fine-tuning plan, Interaction Engine, Memory and Routines for ElevenAgents, early access to ElevenAI, the partner program and Marketplace, and the expanded NVIDIA partnership. The material provides no quantitative evaluation, so the effectiveness of these features has no checkable comparison yet.

Several items remain plans or previews: model fine-tuning is coming in the coming months, Interaction Engine was only previewed, ElevenAI is in free early access, and Marketplace purchasing starts next year. The accuracy gain for Scribe v3 Medical is described by the company, and the material gives no specific figures or test sets.