Geo-distributed LLM serving stops reading only grid carbon: counting batching and queueing cuts emissions up to 60% at about 3% SLO violation

A geo-distributed LLM inference service can meet per-token latency targets while sending more requests to datacenters whose grids are cleaner, cutting operational carbon emissions. Earlier carbon-aware scheduling studies mostly placed requests on 15-minute to 1-hour intervals and assumed latency was load-independent, without modeling continuous batching or engine queueing. Across four regions with four datacenters each and four nodes of 8 A100 GPUs per datacenter, on Llama-3.1-8B, Qwen2.5-7B and Mixtral-8x7B under MLPerf per-token latency targets, the mixed-integer linear program scheduler reduced carbon emissions by up to 60% relative to round-robin, with an average SLO violation of 3%.

Before choosing a datacenter for a request, the scheduler estimates the queueing wait, prefill time and per-token decode time that request would see there, then picks the lowest-carbon datacenter among those that still meet the latency targets. Those estimates come from a hardware-calibrated geo-distributed inference simulator that models continuous batching, queueing when an engine lacks memory, inter-region transfer delays, and separate power for the prefill and decode phases. The simulator was validated end-to-end against real vLLM inference within a single node: over 19 arrival rates from 2 to 50 requests per second and more than 40K requests, TPOT error averaged below 5%, TTFT error 9.1%, and GPU energy error stayed within 2.5%.

Pricing the energy of a batch of requests as a shared per-datacenter cost tracks real emissions better than summing per-request costs, and pays off most on the mixture-of-experts model. Per-request integer linear programs treat the carbon cost of requests in the same batch as independent, double-counting the fixed energy of waking an engine. On Mixtral-8x7B, the facility mixed-integer linear program saved up to 8.6 percentage points more carbon than those policies at a 1.3% SLO violation.

The findings apply to cross-region online LLM inference with per-token latency targets, evaluated on Llama-3.1-8B, Qwen2.5-7B and Mixtral-8x7B at low, moderate and high load under MLPerf latency targets. The simulator was validated end-to-end against real inference within a single node for Llama-3.1-8B only, since end-to-end data-parallel dispatch needs several engines, whereas a single Mixtral-8x7B engine at TP=4 already fills the four-GPU node. The authors propose extending the study to multi-turn sessions, where prefix caching couples consecutive requests to the datacenter holding their KV cache.

The latency and power models are calibrated on A100 GPUs, and other accelerators would shift the absolute carbon and latency values. Workloads are independent single-turn requests, so multi-turn sessions are not covered. Output length is unknown at scheduling time, and replacing it with a lightweight predictor moved carbon savings by at most 4.1 percentage points and raised SLO violations by at most 1.1 points. SLO violations rise noticeably once the polling period exceeds 2 seconds.