ServeTwin couples specification-driven analytical timing with a stateful serving loop, reproducing the throughput-interactivity frontier at 3.6% mean error without target-hardware profiling
Related research and updates
Synopsis
The authors present ServeTwin, a closed-loop simulator that couples specification-driven analytical timing with a stateful serving loop, uses iSTAGE to derive per-iteration traces from model and engine specifications while avoiding target-hardware operator profiling, and implements a vLLM-compatible interface that runs unmodified serving benchmarks; against real deployments it reproduces InferenceX's steady-state throughput-interactivity frontier with a 3.6% mean error and predicts LMBenchmark's multi-turn performance with a 9.9% error while tracking KV-cache evolution, and pre-silicon sweeps show that the preferred HBM bandwidth-capacity tradeoff reverses across workload states and that increasing concurrency shifts the bottleneck from memory bandwidth to scheduler and runtime overheads.
Figure 2. The closed-loop simulation architecture of ServeTwin. Simulated execution times from iSTAGE and ASTRA-sim dynamically dictate subsequent scheduling decisions in a continuous feedback loop.
arXivInterpretation
ServeTwin couples specification-driven analytical timing with a stateful serving loop, capturing feedback among request completion, scheduling, queue state, and KV-cache evolution, including prefill-decode disaggregation and multi-turn agent workloads. Existing simulators provide only subsets of the capabilities needed for realistic design exploration, lacking the combination of stateful closed-loop execution, timing prediction without profiling target hardware, and direct execution of unmodified serving benchmarks. The abstract states this coupling design as a capability description and contrast, without internal implementation details or ablation data.
iSTAGE, an analytical trace generator, derives per-iteration traces from model and engine specifications while representing ragged batches and discrete engine mechanisms such as CUDA-graph batch padding, letting ServeTwin avoid target-hardware operator profiling. Shifting timing prediction from dependence on target-hardware profiling to specification-derived traces is the key difference from the capability subsets of existing simulators. The abstract gives a mechanism-level description but no separate error or overhead figures for iSTAGE.
ServeTwin decomposes execution time into component-owned throughput, scheduler, and runtime costs, letting unaffected parameters transfer across platforms and confining recalibration to changed hardware or software components. This ownership-based decomposition makes cross-platform transfer and localized recalibration a design goal rather than requiring whole-system refitting. The abstract states this as design intent and gives no quantitative validation of cross-platform transfer.
Against real deployments, ServeTwin reproduces InferenceX's steady-state throughput-interactivity frontier with a 3.6% mean error and predicts LMBenchmark's multi-turn performance with a 9.9% error while tracking KV-cache evolution; pre-silicon sweeps show the preferred HBM bandwidth-capacity tradeoff reverses across workload states and that increasing concurrency shifts the bottleneck from memory bandwidth to scheduler and runtime overheads. Combining benchmark validation with pre-silicon design sweeps lets distributed LLM serving systems be explored before target hardware is available. The abstract reports the two error figures of 3.6% and 9.9% but does not state sample sizes, repetition counts, or the error definition.
Perspective
The work targets researchers and systems engineers who need to explore distributed LLM serving architectures before target hardware is available, covering steady-state throughput-interactivity tradeoff analysis, multi-turn agent workload prediction, and design sweeps involving prefill-decode disaggregation and KV-cache evolution. Its interface is vLLM-compatible and runs unmodified serving benchmarks, so it fits into existing serving-benchmark workflows as a replacement for or complement to physical-cluster experiments. The validation described in the abstract concerns two comparisons, InferenceX and LMBenchmark, and the pre-silicon sweep conclusions concern the HBM bandwidth-capacity tradeoff and concurrency-related bottleneck shifts, with applicability bounded by the workload states described in the abstract.
The abstract does not state the precise definition, statistical basis, or repetition count behind the 3.6% and 9.9% errors, nor does it give an accuracy or overhead evaluation of iSTAGE itself. The component-owned decomposition is claimed to let unaffected parameters transfer across platforms, but the abstract provides no quantitative validation of cross-platform transfer. The pre-silicon sweep conclusions about the reversal of the HBM bandwidth-capacity tradeoff and the bottleneck shift depend on the workload states described in the abstract, and their exact boundaries remain to be confirmed in the body. Because this reading is based on the abstract only, figures and body details are not included, and the reproducible conditions for the reported numbers and mechanisms remain open questions to verify.