Nebula lifts position-independent caching to the cluster: median time-to-first-token falls 61.4% on average across four models and four workloads
Synopsis
For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.
Interpretation
Nebula is the first distributed LLM serving framework to treat position-independent cache segments as first-class cluster cache objects that can be independently indexed, placed, transferred, reused or computed across nodes. Distributed caching systems such as Mooncake, LMCache and MemServe reuse key-value cache across nodes but still rely on strict prefix matching; PIC methods such as CacheBlend and EPIC enable position-independent reuse only within a single worker and take no part in cluster-wide resource management. On a cluster with seven prefill workers and one decode worker, placing PIC-enabled workers behind Dynamo's prefix-aware router still left cache hit ratio 10.6 to 56.7 percentage points below the ideal PIC upper bound across six workloads, reaching only 33.4% against 90.0% on ChatRAG-QuAC.
The system maintains its cluster view by combining the router's immediate accounting with worker event feedback, and uses a unified prefill completion-time estimate to choose compute placement and parallelism. Router-side speculation alone reports invalid cache hits, worker-side confirmation alone repeats computation of the same segment, and optimizing cache locality or load balance alone does not necessarily shorten prefill completion time. Replaying RAGPulse and -bench-Retail at 40 QPS with Qwen3.5-35B-A3B on a 7P1D cluster, it reduces invalid hit tokens by 98.4% and 100% relative to optimistic accounting, and repeated computation tokens by 96.1% and 95.1% relative to event-driven updates.
The split of shared cache capacity between prefix caching and PIC is tuned automatically from cluster-wide ghost-hit feedback, which counts evicted entries that are requested again. Unified LRU can let one cache type crowd out frequently reused content of the other, while fixed quotas leave capacity idle on one side as demand shifts; the two policies each win on one of ChatRAG and RAGPulse. On RAGPulse at 48 QPS it raises cache hit ratio by 4.32 and 4.88 percentage points and lowers median time-to-first-token by 11.5% and 14.7% relative to unified LRU and fixed partition; on ChatDev at 12 QPS the hit-ratio gains are 5.48 and 7.37 percentage points.
Perspective
Evaluation covers four models and four public RAG and agent workloads on a deployment of seven prefill workers plus one decode worker with one GPU per worker, replaying requests in original order with Poisson arrivals and limiting each request to one output token; the system focuses on prefill optimization and leaves the decode path unchanged.
The cache-capacity experiment uses the first 800 requests as warmup; the update window is tested over 8, 16, 32 and 64 requests, and the chosen default stays within 1 ms of the best tested window on both RAGPulse and ChatDev; unconfirmed cache-location entries use a 5 s TTL.