# LLM Inference

Large language models store part of their factual knowledge in hash tables keyed by preceding token sequences, and low-bit quantization erases long-tail facts first, while giving more bits to frequently accessed rows does worse than choosing at random. GateRQ stores each row as a ladder of residual codes, computes each lookup's gate from a 1-bit code, and reads deeper levels only for the lookups the gate ranks highest. On DeepSeek-V4.1-Flash it cuts table memory from 23.6 to 3.04 GiB per GPU and raises vLLM throughput over FP8 tables in GPU memory by about 11% and about 31% on the two workloads, losing about 1 PopQA point at 1.50 bits per lookup.

Large language models store part of their factual knowledge in hash tables keyed by preceding token sequences, and low-bit quantization erases long-tail facts first, while giving more bits to frequently accessed rows does worse than choosing at random. GateRQ stores each row as a ladder of residual codes, computes each lookup's gate from a 1-bit code, and reads deeper levels only for the lookups the gate ranks highest. On DeepSeek-V4.1-Flash it cuts table memory from 23.6 to 3.04 GiB per GPU and raises vLLM throughput over FP8 tables in GPU memory by about 11% and about 31% on the two workloads, losing about 1 PopQA point at 1.50 bits per lookup.

Fireworks has added adaptive speculative execution to FireOptimizer, training the draft model for speculative decoding on a customer's own inference workload instead of generic conversation data. Speculative decoding lets a smaller draft model predict several tokens in parallel for the main model to verify, so the draft's hit rate sets the payoff; generic drafts hit poorly on specialized workloads. In one specialized case a generic draft hit 29% and raised latency 1.5x, while FireOptimizer reached a 76% hit rate and 2x speedup, up to 3x over a generic draft model.

Fireworks has added adaptive speculative execution to FireOptimizer, training the draft model for speculative decoding on a customer's own inference workload instead of generic conversation data. Speculative decoding lets a smaller draft model predict several tokens in parallel for the main model to verify, so the draft's hit rate sets the payoff; generic drafts hit poorly on specialized workloads. In one specialized case a generic draft hit 29% and raised latency 1.5x, while FireOptimizer reached a 76% hit rate and 2x speedup, up to 3x over a generic draft model.

BLIS is a deterministic, CPU-only discrete-event simulator that models the whole LLM inference serving pipeline, including admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption, reaching 6.7% median end-to-end error across 36 experiments while running about 200 times faster than real execution.

BLIS is a deterministic, CPU-only discrete-event simulator that models the whole LLM inference serving pipeline, including admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption, reaching 6.7% median end-to-end error across 36 experiments while running about 200 times faster than real execution.

TokenSpeed released an open-source LLM inference engine for agentic workloads that separates the control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to constrain request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Earlier inference engines typically coupled scheduling with execution and checked resource safety at runtime. The engine claims TensorRT-LLM-level performance and vLLM-level usability, and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300.

TokenSpeed released an open-source LLM inference engine for agentic workloads that separates the control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to constrain request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Earlier inference engines typically coupled scheduling with execution and checked resource safety at runtime. The engine claims TensorRT-LLM-level performance and vLLM-level usability, and reports 685 tokens/s per user serving Qwen3.8-Flash-Next on NVIDIA GB300.

TokenSpeed is an open-source LLM inference engine for agentic workloads, backed by the non-profit LightSeek Foundation, aiming at TensorRT-LLM-level performance with vLLM-level usability in production agentic serving. It separates the scheduling control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to enforce safe reuse of request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Kernels are treated as a modular subsystem decoupled from the core engine through a portable public API, a centralized registry, and a plugin mechanism, and the project states it includes one of the fastest MLA implementations on Blackwell for agentic workloads.

TokenSpeed is an open-source LLM inference engine for agentic workloads, backed by the non-profit LightSeek Foundation, aiming at TensorRT-LLM-level performance with vLLM-level usability in production agentic serving. It separates the scheduling control plane from the execution plane: the control plane is a C++ finite-state machine that uses the type system to enforce safe reuse of request lifecycles and KV cache state at compile time, while the execution plane stays in Python for fast iteration. Kernels are treated as a modular subsystem decoupled from the core engine through a portable public API, a centralized registry, and a plugin mechanism, and the project states it includes one of the fastest MLA implementations on Blackwell for agentic workloads.