An inference engine splits control from execution: a C++ state machine guards KV cache while Python stays editable
Synopsis
TokenSpeed released an open-source LLM inference engine for agentic workloads that separates the control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to constrain request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Earlier inference engines typically coupled scheduling with execution and checked resource safety at runtime. The engine claims TensorRT-LLM-level performance and vLLM-level usability, and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300.
Interpretation
Scheduling and execution in the inference engine are split into two layers, so request lifecycles and KV cache ownership are no longer checked at runtime but guaranteed safe for reuse by the compile-time type system. Earlier inference engines kept control logic and execution logic together, leaving resource safety to runtime checks. When welcoming TokenSpeed into the PyTorch Ecosystem Landscape, the PyTorch Foundation noted it is the first open-source LLM inference engine to separate the control plane from the execution plane.
The control plane is implemented in C++ as a finite-state machine that encodes request lifecycle, KV cache ownership and overlap timing as state transitions, while the execution plane is implemented in Python to lower the cost of changes for researchers and engineers. This division places the correctness guarantees of the core scheduling system and the development velocity of a high-level execution layer in one architecture. The engine targets agentic workloads, claims TensorRT-LLM-level performance and vLLM-level usability, and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300.
Kernels are treated as an independent module, separated from the core engine through a portable public API, a centralized registry and selection model, and a plugin mechanism for heterogeneous accelerators. Kernels are no longer bound to the engine body and can be swapped per accelerator. The engine is backed by the non-profit LightSeek Foundation and has hardware and engineering support from vendors including NVIDIA and AMD.
Perspective
This report is for inference engineers and researchers deploying agentic workloads in production, and covers the engine's architectural layering, kernel subsystem and published performance figures. The material is summary-level and does not give full evaluation settings or reproduction steps.
Performance and usability claims come from the project and vendor blogs, and the material provides no independent third-party comparison; the 685 tokens/s per user figure corresponds to the specific combination of Qwen3.8-Flash-Next and NVIDIA GB300 and does not transfer directly to other models or hardware.