# Layer-wise CPU-GPU Scheduling

When model weights do not fit in a single consumer GPU, on-demand weight transfer takes about 98% of decode time. A layer-wise schedule runs some layers on the CPU while asynchronously moving the next layer's weights to the GPU, hiding the transfer behind CPU computation; on an RTX 5090 desktop, OPT-30B inference runs 3.5 times faster than HuggingFace Accelerate and 6.3 times faster than DeepSpeed Zero-Inference.

When model weights do not fit in a single consumer GPU, on-demand weight transfer takes about 98% of decode time. A layer-wise schedule runs some layers on the CPU while asynchronously moving the next layer's weights to the GPU, hiding the transfer behind CPU computation; on an RTX 5090 desktop, OPT-30B inference runs 3.5 times faster than HuggingFace Accelerate and 6.3 times faster than DeepSpeed Zero-Inference.