Coding agents stop rewriting history: ObsDrop drops stale KV instead, reaching 5.37× throughput at 16 concurrent agents
Synopsis
Coding agents mostly append to their history, so prefix-cache reuse reaches 98–99%, yet text-level context compaction rewrites earlier tokens and forces the whole suffix to be recomputed. ObsDrop moves compaction from the prompt into the KV cache: it keeps the logical token prefix unchanged while releasing the KV of stale tool observations and excluding those positions from attention, implemented in SGLang and Sparse-vLLM. At 16 concurrent agents it reaches 2.45–2.85× throughput with FlashInfer and a 5.37× peak with Triton relative to no compaction, with task quality close to no compaction.
Interpretation
Context compaction for coding agents can stop rewriting the prompt and instead reclaim only the KV entries held by stale tool observations, so the next request still matches the full prefix. Truncation, per-step observation masking and summarization all rewrite history at the text level; once an earlier token changes, prefix matching ends at the first edit and the remaining suffix must be re-prefilled. On uncompacted agent trajectories, successive requests hit the prefix cache 98–99% of the time, showing that the reusable prefix is nearly complete and that rewriting history is what costs.
The design separates logical history from physical KV residency: the prompt and token positions stay as they are, stale observation KV slots are marked dead and returned to the allocator, and those positions are removed in place from the KV index list before attention. The implementation relies on SGLang's RadixAttention cache, which maps logical token sequences to physical KV slots through a radix tree, so residency can change while the matched tokens stay the same; vLLM organizes its cache as hash-identified blocks and cannot release entries inside a cached prefix while keeping it matchable. Evaluation covers two serving engines, SGLang and Sparse-vLLM, and two attention backends, FlashInfer and Triton, with Qwen3-Coder-30B-A3B and Devstral-Small-2-24B on SWE-bench Verified and SWE-bench Multilingual.
The gain grows with concurrency and KV pressure, because a smaller physical footprint keeps more agent prefixes resident and avoids the evict-then-re-prefill cycle. Without compaction the full context fills the KV cache, and once the server evicts, agents re-prefill their whole context and tail latency rises; text-level policies lose reuse whenever they rewrite history. At 16 concurrent agents, the Qwen-Multilingual workload on SGLang retains 89% prefix reuse with about 6 s p90 time to first token, while no compaction falls to 32% reuse and about 80 s; throughput relative to no compaction is 2.45–2.85× with FlashInfer and peaks at 5.37× with Triton.
Perspective
Evaluation is limited to coding-agent settings: the Qwen3-Coder-30B-A3B and Devstral-Small-2-24B models, SWE-bench Verified and SWE-bench Multilingual, the SGLang and Sparse-vLLM engines, and the FlashInfer and Triton attention backends; text-level experiments run on Aurora and KV-level experiments on Polaris.
The batching interval for batched masking is fixed at k′ = 2k as a practical choice, and the optimal interval may vary across trajectories and workloads; serving cost is measured with a normalized model weighted by API prices, and although prices and the cached-to-uncached token ratio may change, the trade-off between compaction and prefix reuse remains. Porting ObsDrop to vLLM requires decoupling block identity from KV residency and is left as future work.