Baseten cuts a 770 GB checkpoint sync from 36 seconds to 15, with a 0.8-second pause
Synopsis
Baseten moved the parent copy of its delta weight sync back onto the GPU, cutting a roughly 770 GB GLM-5.3 checkpoint sync from 36 seconds with a 6-second request pause to 15 seconds with a 0.8-second pause, and raising throughput by as much as 3.2x. The earlier design kept the full parent in host memory, which on an 8×B300 node could not coexist with a 1.1 TB host KV cache, so the cache shrank to 275 GB. The new path reads only changed GPU positions and verifies updates with delta checksums before writing.
Interpretation
Delta weight sync no longer needs a second full copy of the parent checkpoint in host memory; GPU memory becomes the only parent. Previously every replica held the full parent in host RAM, and a roughly 770 GB GLM-5.3 could not coexist with a 1.1 TB host KV cache on an 8×B300 node, so the cache shrank to 275 GB; promoting the CPU copy to the new policy took about 45 seconds per update and blocked the next delta. For the same roughly 770 GB GLM-5.3 checkpoint, sync time fell from 36 seconds to 15 seconds, the request pause fell from 6 seconds to 0.8 seconds, and throughput improved by as much as 3.2x.
The decoder handles only the bytes that change, never materializing the dense XOR tensor and never reading model weights. The old design needed the host-memory parent to turn XOR bytes into new values and had to patch that copy after every update, leaving two sources of truth on GPU and CPU. At startup the model is loaded from a local disk copy, every served tensor is verified against the engine's actual GPU layout with a per-tensor Adler-32 checksum, and the copy is deleted; when a delta arrives the decoder walks zstd commands directly and keeps only the position and XOR mask of each nonzero byte.
Delta checksums can be corrected byte by byte instead of rehashing the whole tensor. The naive check rehashes the entire tensor, so its cost grows with tensor size. Starting from the checksum recorded at startup, one correction per changed byte is computed from its position, old value and new value and compared with the trainer's checksum; the optimization made the whole update up to 2.3x faster, taking a 30 GB Kimi K3 delta from about 230 to about 100 seconds, and a corrupted download or wrong parent is rejected before any GPU write.
The freed host memory goes to KV cache offloading, which pays off for long-prefix agent workloads. At concurrency 16 the KV cache already fits on the GPU, so offloading makes no difference; at concurrency 48 the agents' prefixes no longer fit in GPU memory but do fit in host memory. On the AgentX benchmark with Kimi K3 at concurrency 48, the cache hit rate rose from 2% to 92% and output throughput per GPU rose 3.2x; reloading an evicted 150k-token prefix from host memory takes 0.48 seconds versus 4.9 seconds of fresh prefill.
Perspective
The demonstrated setting is one replica with trainer-shaped fixture deltas, timed from the trainer's delta weight sync request to the replica serving the new policy; the models are a roughly 770 GB GLM-5.3 and a 1.7 TB Kimi K3 running on two GB300 nodes with four GPUs each.
All runs are one replica with trainer-shaped fixture deltas, timed from the S3 marker to the first new response; at concurrency 16 KV cache offloading makes no difference, and the gain appears only where prefixes exceed GPU memory, as at concurrency 48.