NVIDIA open-sources Snapshot: initialized GPU inference workloads become reusable artifacts, cutting replica restore to under 45 seconds
Synopsis
NVIDIA has open-sourced NVIDIA Snapshot, a checkpoint-and-restore component for GPU inference workloads on Kubernetes that captures a fully initialized workload, including CPU and GPU memory, as a portable artifact and restores it on nodes with the same GPU model and driver, skipping weight loading, compilation and graph capture. Across vLLM, SGLang and TensorRT LLM, from a 600-million-parameter model to mixture-of-experts models with 1 TB checkpoints on eight GPUs, every tested configuration restored a replica in under 45 seconds, 8x to 36x faster than a cold start.
Interpretation
Starting an inference replica or recovering it after a failure no longer requires initializing from scratch each time; an initialized GPU workload can be saved and brought back on compatible infrastructure. Previously, when inference demand spiked or a replica failed, available GPUs did not translate immediately into serving capacity, because initializing an LLM workload, known as a cold start, could take minutes and delayed both scaling and recovery. On vLLM, SGLang and TensorRT LLM, covering a 600-million-parameter model through mixture-of-experts models with roughly 600 GB to 1 TB checkpoints across eight GPUs, restore times across all tested configurations were 3.6 to 42.7 seconds, 8x to 36x faster than cold start; the largest MoE models restored in under 43 seconds against cold starts of 6 to 19 minutes.
The snapshot treats the entire initialized process state as one portable artifact, and a restore continues from the checkpoint rather than repeating the initialization sequence. Cold-start time is spread across starting processes and libraries, reading weights and building tensors, memory profiling and KV-cache allocation, and kernel compilation, autotuning and CUDA graph capture; a restore skips these repeated steps. Checkpoints are created through Kubernetes custom resources, and a restore adds the nvidia.com/restore-from annotation to a new Pod naming the PodSnapshot; the restore runs in four stages, agent setup, CRIU reading the process image back, CUDA state copied into GPU memory, and wake/remap, with CUDA restore taking 18 to 29 seconds and CRIU restore 4.2 to 10.8 seconds on the eight-GPU MoE configurations.
The one-time cost of checkpointing pays off with the first restored replica. Checkpointing takes extra time to capture an initialized replica and consumes storage, a cost paid once per model and configuration. In every tested configuration, checkpoint time plus restore time was shorter than a single cold start; checkpoint creation took 15 seconds to 5.5 minutes depending on the model and configuration.
Perspective
The demonstrated scope is compatible workloads on Kubernetes within a single node, spanning multiple GPUs, with restore targets on the same GPU model and driver; measurements exclude pulling the container image and setting up the container runtime, and the KV cache is released before every checkpoint and is not part of the artifact.
Restore speed is set mostly by the storage backend, which constrains the CRIU stage; checkpoint creation time varies from 15 seconds to 5.5 minutes with model and configuration, and workloads must meet Snapshot's checkpoint/restore and runtime compatibility requirements.