NVIDIA prices agentic inference per byte: Vera Rubin NVL72 reaches up to 30x GB300 NVL72's throughput per megawatt

NVIDIA redefines AI-factory memory efficiency as useful tokens produced per byte rather than capacity per GPU. From GB300 to Rubin, capacity held at 288 GB per GPU while bandwidth rose to 19.2 TB/s, so capacity alone cannot explain the agentic throughput gain. On SemiAnalysis AgentX, which replays coding-agent sessions with real tool-call gaps and growing context, Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user, with both at 288 GB per GPU.

Weights and KV cache are compressed into fewer bits while retaining enough accuracy. NVFP4 uses a 4-bit E2M1 element with an FP8 scale shared by every 16 values and a per-tensor FP32 scale on top, finer-grained than naive 4-bit formats. Applied to weights, NVFP4 shrinks the model footprint by about 3.5x versus FP16 and 1.8x versus FP8; across AA-AgentPerf, SemiAnalysis AgentX and MLPerf Endpoints and leading MoE architectures it delivers up to 4x the throughput of an FP8 baseline at the same interactivity; NVFP4 KV cache cuts footprint by up to 50% versus FP8, keeping about 2x more context resident.

MoE expert weights are spread across more GPUs, and KV cache is scheduled as a fleet-wide resource. When experts are packed onto few GPUs, much of each card's memory holds weights and every decode step pays a heavy weight-loading cost for MoE GroupGEMMs. TensorRT-LLM wide expert parallelism at EP64 on DeepSeek-R1 leaves four experts per layer per GPU, 232 expert blocks across 58 MoE layers out of 14,848; on GB200 NVL72, EP32 delivers up to 1.8x more output tokens per second per GPU than EP8 at 100 tokens per second per user; Dynamo uses KV-aware routing, disaggregated prefill and decode, and the KV Cache Runner across GPU, pinned CPU and storage tiers.

CPU memory and pod-level flash become additional KV-cache tiers so context survives tool calls and nodes. A session's KV cache held in HBM during a tool call occupies capacity without producing tokens, and eviction forces a full prefill on return. Each Vera CPU provides up to 1.5 TB of LPDDR5X, more than 5x a single Rubin GPU's HBM4 capacity, linked coherently to Rubin at 1.8 TB/s over NVLink-C2C; CMX delivers up to 5x higher tokens per second and up to 5x better power efficiency than traditional storage for long-context inference.

The conclusions cover NVIDIA's own GB300, GB200 and Vera Rubin NVL72 platforms and the TensorRT-LLM, Dynamo and CMX software stack; the performance figures come from benchmark replays such as SemiAnalysis AgentX and vendor validation tests, not from production deployment measurements.

The 30x throughput per megawatt is an upper figure given at 160 tokens per second per user on a benchmark that replays coding-agent sessions; NVFP4's 4x throughput is measured against an FP8 baseline at the same interactivity; CMX's 5x tokens per second and 5x power efficiency are relative to traditional storage for long-context inference.