No GPUs needed to plan LLM serving: BLIS simulates the full pipeline on CPU and holds a 300 ms TTFT SLO on four H100s

Planning an LLM inference deployment no longer has to occupy a GPU cluster. BLIS moves latency and throughput prediction for a serving deployment onto a CPU, so configuration search and policy iteration can run locally and repeatedly. Confirming the coupled tradeoffs among routing, admission, batching, and KV-cache reuse previously meant measuring them on a GPU cluster: one workload evaluation, 10,000 requests at 10 requests per second served by Llama-2-70B on NVIDIA H100 nodes, takes roughly 18 minutes of sustained GPU time plus provisioning, download, and warm-up, and a meaningful exploration repeats this across hundreds of configurations, policies, and workloads. Across 36 experiments spanning six models, three GPU types, and four workloads, BLIS reaches 6.7% median end-to-end error and runs about 200 times faster than real execution; median error is 5.5% on H100 and 15.7% and 13.3% on the unprofiled A100 and L40S.

BLIS advances the whole serving pipeline as interacting components rather than estimating stages separately. A request moves through admission, routing, queueing, batch formation, prefill and decode execution, KV-cache allocation, and preemption on one event timeline. Existing predictive tools each cover a fragment of the pipeline: closed-form estimators ignore continuous batching, KV-cache pressure, and queueing, mispredicting by 20 to 90 percent under load, while profiled-operator simulators are more faithful but need a fresh calibration run for each new model, GPU, or tensor-parallel degree. The latency model pairs roofline-style basis functions with scaling coefficients and overhead terms fit once, jointly across 15 real-system H100 experiments covering six model families, dense and mixture-of-experts architectures, and tensor-parallelism degrees of 1, 2, and 4; the six evaluation models are disjoint from the fitting set. The simulator advances by event and produces byte-identical output for a given seed.

The simulator targets researchers and engineers who need to compare serving configurations and policies before committing GPU capacity, and its supported settings are configuration search and policy evolution: sweeping hundreds of candidates on a CPU, then validating a small set on a cluster. Latency-model training and accuracy validation are single-instance, with each experiment measuring one vLLM instance of up to 8 GPUs under tensor parallelism; the configuration search and both policy case studies instead run multi-instance on real llm-d deployments of up to 8 H100 GPUs across replicas, with the admission policy validated on 4 H100s and the flow-control policy on 2 H100s. The case-study validations measure relative policy improvement rather than absolute latency.

Absolute prediction error at multi-node scale remains to be measured, where cross-node interconnect and placement effects come into play. Communication is reduced to a bandwidth-based analytical term: topology is invisible, each transfer is assumed to see full bandwidth, so NVLink contention is under-estimated at higher tensor-parallel degrees and would diverge further on shared cross-node links, and latency variability is absent. Expert parallelism, multi-adapter (LoRA) serving, disaggregated prefill/decode with cross-pool KV-cache transfer, speculative decoding, and KV-cache compression are not modeled. Evaluation covers dense and mixture-of-experts transformers on NVIDIA GPUs; architectures that replace the KV cache with recurrent state or reshape the per-token cache, such as state-space models, RWKV, and multi-head latent attention, may weaken generalization, and non-NVIDIA accelerators are untested. The simulator mirrors the scheduling semantics of a specific vLLM version and the distributed inference logic of llm-d, so it must be updated as those implementations evolve. The latency model is calibrated for the region where the system is loaded but not deeply saturated, and under heavy overload no deterministic simulator can predict queueing effects precisely.