A 30B model won't fit on one consumer GPU: run some layers on the CPU and hide the weight transfers behind that work
When model weights do not fit in a single consumer GPU, on-demand weight transfer takes about 98% of decode time. A layer-wise schedule runs some layers on the CPU while asynchronously moving the next layer's weights to the GPU, hiding the transfer behind CPU computation; on an RTX 5090 desktop, OPT-30B inference runs 3.5 times faster than HuggingFace Accelerate and 6.3 times faster than DeepSpeed Zero-Inference.