Yifei Liu

For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.

For retrieval-augmented generation and agent workloads, Nebula promotes position-independent caching (PIC) segments, which let the same text segment reuse already-computed key-value cache under a different prefix, into cluster-level cache objects, enabling remote reuse and parallel prefill. Distributed caching systems had reused key-value cache only under strict prefix matching, while single-node PIC could not draw on other nodes' cache or compute. Across four models and four real RAG and agentic workloads, Nebula cuts median time-to-first-token by an average of 61.4% against the strongest baseline and recovers an average of 76.1% of the tokens missed by prefix caching.