An inference engine splits control from execution: C++ state machine owns the KV cache, Python stays for researchers

The scheduling control plane and the execution plane of an LLM inference engine are split into two layers, with the control plane written as a C++ finite-state machine that describes request lifecycles and KV cache ownership, and the execution plane left in Python. Earlier inference engines mixed control and execution, so resource safety relied on runtime checks and researchers changing scheduling logic had to work against low-level implementation details. When welcoming TokenSpeed into the PyTorch Ecosystem Landscape, the PyTorch Foundation listed being the first to separate the control plane from the execution plane as an architectural differentiator.

Safe reuse of shared resources such as the KV cache is enforced by the type system at compile time rather than checked at runtime. Runtime checks expose errors only after a request is already running, whereas compile-time constraints move that class of error to the build stage. Request lifecycle, KV cache ownership, and overlap timing are encoded as a finite-state machine, with safe KV resource reuse enforced by the type system at compile time.

Kernels are pulled out of the core engine into a pluggable, layered subsystem that connects to heterogeneous accelerators through a public API, a centralized registry, and a plugin mechanism. When kernels are coupled to the engine, swapping hardware or implementations means touching core code; a public API plus a registry separates selection from implementation. The project states the kernel system includes one of the fastest MLA implementations on Blackwell for agentic workloads.

The engine is backed by the non-profit LightSeek Foundation rather than a commercial company, with hardware and engineering support from vendors including NVIDIA and AMD. An engine led by a commercial company is constrained by that company's own interests in its roadmap choices, and the non-profit backing is presented as the source of neutrality. The project describes this neutral position as the Switzerland of open-source LLM inference engines and lists support from NVIDIA and AMD.

This report is for researchers and engineers working on inference serving and agentic systems, and it describes the engine's architectural layering, kernel organization, and ecosystem position. The public material places its performance statements in agentic-workload settings, listing entries such as 685 tokens/s per user when serving Qwen3.8-Flash-Next on NVIDIA GB300 and 580 TPS on Qwen3.5-397B-A17B. The material also presents Pareto curves against TensorRT-LLM on agentic workloads involving Kimi K2.5 and B200.

The material is presented as a summary, and the architecture description and performance entries come from official releases and vendor blogs without reproducible experimental settings. The performance statements are limited to agentic workloads and the listed hardware and model combinations, and behavior on other workload types or accelerators is not stated. The material gives no quantified account of the cross-language call overhead that separating the control and execution planes may introduce.