EAGLE-4 stops fixing its verification tree: GPU-measured costs set tree shape, reaching 1.85x throughput on creative prompt expansion
Synopsis
EAGLE-4 replaces a fixed-depth speculative decoding tree with one that widens or deepens level by level using measured GPU verification cost, paired with a lightweight drafter. On NVIDIA B300 with tensor parallelism 4, it reaches 660.6 tokens/s on Fireworks Bench and 622.9 on Prompt Enhancement SpecBench at batch size 1, or 1.63x and 1.85x the fastest alternative; at batch size 8 the advantages are 1.36x and 1.62x. Creative prompt expansion is harder for drafters to predict, yet the tree recovers more usable continuations from that uncertainty.
Interpretation
The verification tree no longer has a fixed depth or candidate count; at runtime it decides level by level whether to widen or deepen, based on measured GPU verification cost. Chain verification commits to one continuation per position, so an early mismatch discards every later token, while a fixed tree requires preset depth and top-k settings. On NVIDIA B300 with TP=4, at a verification budget of 16 tokens, the tree raises acceptance length by 21.1% on the Fireworks dataset and 26.1% on the fal text-only subset relative to the chain.
The throughput gain is larger on creative prompt expansion, where drafting is harder. At the same verification budget, both methods show lower acceptance length on the fal creative prompts than on the Fireworks dataset, marking these workloads as harder for the drafters. At batch size 1, EAGLE-4 Harness reaches 660.6 tokens/s on Fireworks Bench and 622.9 on Specbench, 1.63x and 1.85x the fastest alternative; at batch size 8 it reaches 3566.4 and 3551.9 tokens/s, advantages of 1.36x and 1.62x.
The tree's gain concentrates in early positions, which are more likely to be reached. A long chain spends more of its verification budget on later positions that are less likely to be reached. On the Fireworks dataset, comparing the tree at L=8 with the chain at N=7, the first-token hit rate rises from 72.5% to 85.3%, an increase of 12.8 percentage points.
Perspective
The demonstrated setting is two evaluation sets on NVIDIA B300 with tensor parallelism set to 4: 138 Fireworks prompts and 200 text-only prompts from the fal prompt expansion data, with throughput compared at batch sizes 1 and 8. The release also includes Prompt Enhancement SpecBench, 400 examples drawn from 18 datasets across text-to-video, image-to-video, text-to-image and image-to-image, 100 per category.
The released Qwen 3.8 27B model does not yet include multiple draft layers, mixture-of-experts support or Latent ReLU, which the release expects future models to use. Creative prompt expansion has open-ended continuations that are harder for drafters to predict: acceptance length is 2.66 in the highest-entropy quartile versus 2.97 in the lowest.