Respan rewrites its AI gateway in Rust with a tiny core, so every provider and feature is a plugin
Synopsis
Respan released an open-source Rust AI gateway that makes providers, protocols, modalities and reliability features all plugins, leaving the core to dispatch events. The team had wrapped LiteLLM in more than 100,000 lines of code, and after a routine upgrade caused intermittent connection errors and a rollback whose cause is still unknown, it rebuilt the gateway. Adding a provider now takes a short file, 22 lines for Mistral, with each quirk tested separately.
Interpretation
The new gateway moves providers, protocols, modalities and gateway features out of the core, which only dispatches events, so teams can assemble the AI infrastructure their application needs. Since 2024 the team had wrapped LiteLLM layer by layer, and the wrapper code at one stage exceeded 100,000 lines, while each organization needed different API key settings, routes and limits. In January a routine LiteLLM upgrade caused intermittent connection errors; the team rolled back, still has not found the cause, and now backports upstream fixes manually instead of upgrading directly.
An adapter hands each HTTP or WebSocket connection to the first client format that recognizes it, and that format exposes the request as a two-way stream of typed input and output events that the core runs plugins over in registration order. Plugins can choose a provider, filter or wrap either stream, make additional calls, return a response or refuse the request, after which a provider format opens the upstream connection. When client and provider share a format and no plugin changes the payload, the original bytes and protocol-specific fields pass through unchanged; when formats differ, shared types translate messages, tool calls, results, streams and usage.
Perspective
The release targets engineering teams that call several models from a single endpoint and want to extend features themselves. The gateway covers protocols including chat, realtime voice and MCP, modalities including images and video, and example plugins for smart routing, fallbacks and PII redaction. An application built on the OpenAI Realtime API can use Gemini Live through the gateway, though not every feature translates.
The team says it has not determined what caused the connection errors after that LiteLLM upgrade. Cross-format translation is not fully equivalent, and some features cannot be translated between client and provider. The release gives no performance or stability measurements for the new gateway.