# KAI Scheduler

NVIDIA has donated Agent Substrate, an open-source runtime for running large numbers of sandboxed AI agents on Kubernetes, to the CNCF Sandbox. Agents spend most of their time waiting on a prompt, a tool result or a model response; keeping one Pod per session holds memory for idle sessions, while creating Pods on demand pays a multi-second cold start at every step and loses state. Agent Substrate multiplexes many sessions over a small pool of warm worker Pods, checkpointing idle sessions including memory and filesystem and restoring them onto whichever worker is free, and the project reports 10x higher density than standard containers.

NVIDIA has donated Agent Substrate, an open-source runtime for running large numbers of sandboxed AI agents on Kubernetes, to the CNCF Sandbox. Agents spend most of their time waiting on a prompt, a tool result or a model response; keeping one Pod per session holds memory for idle sessions, while creating Pods on demand pays a multi-second cold start at every step and loses state. Agent Substrate multiplexes many sessions over a small pool of warm worker Pods, checkpointing idle sessions including memory and filesystem and restoring them onto whichever worker is free, and the project reports 10x higher density than standard containers.

NVIDIA CUDA 13.4 adds an application-transparent time-sharing mode that confines each workload to a fraction of GPU memory, so an allocation above the requested size fails as it would on a smaller GPU. KAI Scheduler offers an NvFractions sharing mode from v0.18.0 that picks GPU slices and writes memory and compute mode into Pod annotations, and the new open-source gpu-fractioning operator enforces those limits on each node and measures per-fraction use.

NVIDIA CUDA 13.4 adds an application-transparent time-sharing mode that confines each workload to a fraction of GPU memory, so an allocation above the requested size fails as it would on a smaller GPU. KAI Scheduler offers an NvFractions sharing mode from v0.18.0 that picks GPU slices and writes memory and compute mode into Pod annotations, and the new open-source gpu-fractioning operator enforces those limits on each node and measures per-fraction use.

NVIDIA introduces Topograph, an open source toolkit whose providers discover cluster topology from cloud APIs or on-premises fabric systems and normalize it into a canonical model, while engines translate that model into Kubernetes node labels, NFD resources, Slinky ConfigMaps, Slurm topology configuration, or topology JSON, letting schedulers on Kubernetes, Slurm, and Slinky make proximity-aware placement decisions and supporting topology-aware gang scheduling through KAI Scheduler and Kueue.

NVIDIA introduces Topograph, an open source toolkit whose providers discover cluster topology from cloud APIs or on-premises fabric systems and normalize it into a canonical model, while engines translate that model into Kubernetes node labels, NFD resources, Slinky ConfigMaps, Slurm topology configuration, or topology JSON, letting schedulers on Kubernetes, Slurm, and Slinky make proximity-aware placement decisions and supporting topology-aware gang scheduling through KAI Scheduler and Kueue.