# TokenSpeed

TokenSpeed released an open-source LLM inference engine for agentic workloads that separates the control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to constrain request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Earlier inference engines typically coupled scheduling with execution and checked resource safety at runtime. The engine claims TensorRT-LLM-level performance and vLLM-level usability, and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300.

TokenSpeed released an open-source LLM inference engine for agentic workloads that separates the control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to constrain request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Earlier inference engines typically coupled scheduling with execution and checked resource safety at runtime. The engine claims TensorRT-LLM-level performance and vLLM-level usability, and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300.

TokenSpeed is an open-source LLM inference engine for agentic workloads, backed by the non-profit LightSeek Foundation, aiming at TensorRT-LLM-level performance with vLLM-level usability in production agentic serving. It separates the scheduling control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to enforce safe reuse of request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Kernels are treated as a modular subsystem decoupled from the core engine through a portable public API, a centralized registry, and a plugin mechanism, and the project states it includes one of the fastest MLA implementations on Blackwell for agentic workloads.

TokenSpeed is an open-source LLM inference engine for agentic workloads, backed by the non-profit LightSeek Foundation, aiming at TensorRT-LLM-level performance with vLLM-level usability in production agentic serving. It separates the scheduling control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to enforce safe reuse of request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Kernels are treated as a modular subsystem decoupled from the core engine through a portable public API, a centralized registry, and a plugin mechanism, and the project states it includes one of the fastest MLA implementations on Blackwell for agentic workloads.