GPUs stop being handed out whole: CUDA 13.4 enforces a memory boundary, and Kubernetes carves cards into fractions
Synopsis
NVIDIA CUDA 13.4 adds an application-transparent time-sharing mode that confines each workload to a fraction of GPU memory, so an allocation above the requested size fails as it would on a smaller GPU. KAI Scheduler offers an NvFractions sharing mode from v0.18.0 that picks GPU slices and writes memory and compute mode into Pod annotations, and the new open-source gpu-fractioning operator enforces those limits on each node and measures per-fraction use.
Interpretation
A Kubernetes cluster can now share one GPU by memory fraction instead of giving an entire device to a single workload. Previously MIG split a GPU into hardware-isolated instances, time-slicing let processes take turns without memory enforcement, and MPS let them run concurrently in separate address spaces. CUDA 13.4 adds a time-sharing mode with a memory boundary alongside the existing sm-sharing compute mode; an allocation above the requested size fails, and each workload sees only a fraction of GPU memory.
Choosing a fraction and enforcing it are split into two jobs: the scheduler decides which slice of which GPU a workload gets, and a node-side operator applies the limits to the device. That node-side work is independent of the scheduler, so any Kubernetes scheduler that can write these annotations can enforce GPU fractions through CUDA. KAI Scheduler offers NvFractions from v0.18.0, recording each container's memory allocation and compute-sharing mode in Pod annotations, which gpu-fractioning reads to configure the assigned GPUs.
Memory can be requested as an absolute size or as a portion of a GPU, with an optional limit the workload may grow into when memory is free. When the limit is omitted, it equals the request and the workload is capped at exactly what it asked for. Absolute sizes use annotations such as nvidia.com/container.<c>.gpu-memory.limit, portions use kai.scheduler/container.<c>.gpu-memory.portion.limit, and a portion limit must be greater than the gpu-fraction and less than 1.
Compute sharing comes in two modes, differing in whether workloads take turns or run together. Time-slicing is serial and gives each workload its own error domain; sm-sharing is parallel, allocates streaming multiprocessors on demand, and puts all workloads in one error domain. With n workloads sharing a GPU, each gets roughly 1/n of the compute in either mode; a shared GPU runs one compute mode at a time, and KAI Scheduler tracks this per GPU rather than per node.
Perspective
The scope is NVIDIA GPU nodes running Kubernetes, with the NVIDIA GPU Operator providing the driver and container runtime setup and a scheduler writing per-container annotations; KAI Scheduler offers NvFractions from v0.18.0, and administrators choose one sharing mode per cluster.
Under sm-sharing all workloads share a single error domain, and the material advises using it only for workloads trusted to run together; under time-slicing each workload has its own error domain. A shared GPU runs one compute mode at a time, so the scheduler must avoid mixing modes on the same GPU.