Async RL squeezes context into fewer decode steps; INT4 KV cache speeds rollouts but loses accuracy, bounded INT8 protection recovers it

Asynchronous scheduling removes the long-tail idle time but packs the same context processing into fewer decode steps, so each step carries more KV work and the rollout bottleneck shifts from weight reads to the KV cache. Earlier low-precision rollout mostly quantized weights or activations, with KV caches kept at FP8 or BF16; under sustained concurrency weight reads are amortized while attention still processes every response's context. In matched-workload replay, KV4 reaches about 3.6x, 3.4x, and 3.1x decode speedup at the early, middle, and late checkpoints, KV8 about 2.1x, while W4A16 and W8A8 stay near or below BF16 speed.

Using an INT4 KV cache directly degrades learning even with token-level truncated importance sampling, so sampler fidelity must be recovered without giving up INT4's execution and capacity gains. 8-bit KV rollout tracks BF16 closely, whereas 4-bit KV rollout misses BF16's later improvement and finishes 5.24 percentage points lower on MATH-500 avg@8. BF16, KV8, and KV4 trajectories are compared under the same asynchronous GRPO and TIS recipe on Qwen3-8B-Base, and shared-prefix conditional KL at identical BF16 prefixes shows KV4-RL cutting that mismatch by about 60% relative to KV4.

KV4-RL keeps the whole context in paged INT4 and retains INT8 copies only for a fixed set of attention-sink and recent positions, executed through an integer attention path and an exact regional softmax merge. Raising precision across the entire cache would add overhead that grows with context length, whereas the protected set holds at most 144 positions and its extra storage and attention work stay bounded. Full-parameter GRPO training runs on Qwen3-4B-Base and Qwen3-8B-Base for 600 and 400 updates, collecting 128 groups of eight responses per iteration with a 16,384-token response cap.

End to end, KV4-RL accelerates rollout and full iterations while keeping the generated workload close to BF16, with larger gains as responses grow longer. Over 400 updates at 8B it reaches about 2.3x cumulative rollout speedup and about 1.6x full-iteration speedup; over 600 updates at 4B the figures are about 1.7x and 1.3x, while run-wide mean response lengths are 1.45k tokens for BF16 and 1.46k for KV4-RL. Under a common BF16 evaluator, KV4-RL's six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout at both model sizes, while KV4 falls 4.6 points below BF16 at 8B and 0.8 points at 4B.

The results cover two dense Qwen3 models, mathematical reasoning data, and a colocated asynchronous GRPO configuration, in the long-generation, high-concurrency regime where KV concentration is most visible. Evaluation uses BF16 weights and KV caches, so reported accuracy measures the learned policy rather than quantized inference output. Fixed-workload replay isolates execution speedup at a matched decode state, while end-to-end timing also includes the higher concurrency enabled by a smaller cache; the two contributions are not separated.

KV4-RL's advantage grows with response length, so the shorter 4B responses yield smaller speedups. At the late 8B checkpoint, asynchronous decode throughput falls back to the synchronous level, indicating that a fixed page budget limits the scheduling benefit. KV4's larger 8B deficit appears in the longer-context regime, but that comparison does not separate response length from model size. Wider protection keeps reducing mismatch at a progressively larger cost in decode speedup, and the default is a balance within that trade-off.