Speculative decoding drafters get a new training target: one D-REAL epoch lifts DFlash2 serving throughput 9.4% on GSM8K
Synopsis
Speculative decoding drafters are usually trained with token-level cross-entropy, yet at fixed overhead end-to-end speedup is governed by acceptance length, the number of draft tokens the target model accepts consecutively. D-REAL writes that acceptance length as an expectation along the drafter's own sampled trajectories and builds a differentiable surrogate from acceptance ratios with stop-gradient denominators, giving an unbiased gradient estimator. Across ten benchmarks on the parallel drafters DSpark and DFlash2, one post-training epoch raises macro-average acceptance length from 5.07 to 5.13 and from 4.88 to 5.10; in concurrent GSM8K serving, DFlash2 output throughput rises 9.4%.
Interpretation
The drafter's training objective changes from a token-level loss to direct optimization of the single-round expected acceptance length, the number of draft tokens the target model accepts consecutively in one verification round. Cross-entropy and KL losses add position-wise terms and do not explicitly optimize the length of the accepted prefix; D-PACE assigns positional weights from an acceptance-length proxy built on reference-token confidence. Across 40 settings, two drafter architectures, ten benchmarks and two decoding temperatures, D-REAL exceeds D-PACE in 37, ties in 1 and is lower in 2; macro-average acceptance length rises from a 5.07 baseline to 5.13 on DSpark and from 4.88 to 5.10 on DFlash2.
The implementation computes position-wise overlap between target and draft distributions along prefixes the drafter itself sampled, weighted by prefix survival probability. Earlier positional weights came from confidence proxies on offline reference sequences rather than the rejection-sampling expectation over actual draft-sampled trajectories. Training samples anchor positions uniformly along offline target-generated sequences and calls the frozen target model online to score the draft-sampled prefix; on an 8-GPU Hopper node, one D-REAL post-training epoch takes about the wall-clock time of one baseline epoch.
Gradients keep the derivative of the proposal distribution itself through stop-gradient behavior-proposal denominators, so the estimator carries both the acceptance-ratio term and the score-function term of the trajectory. Detaching weights alone does not establish whether an estimator is biased, while detaching the whole importance-ratio product would discard the entire derivative of the prefix survival factor. Under locally invariant proposal support and almost-everywhere differentiability, proposal probabilities cancel term by term against their denominators, yielding an unbiased stochastic gradient for expected acceptance length; in ablation, replacing the dynamic prefix with static exponential decay gives DSpark average acceptance length 5.23 at temperature 0 and 4.91 at temperature 1, below the full D-REAL values of 5.26 and 4.96.
Perspective
The demonstrated setting is Qwen3-8B as the frozen target model with two 5-layer parallel drafters, DSpark and DFlash2, evaluated on ten benchmarks spanning math, code and chat at greedy and stochastic temperatures, plus end-to-end serving in SGLang on GSM8K at a given concurrency.
Online target verification adds per-step training latency, although one post-training epoch takes about the wall-clock time of one baseline epoch; training anchors are still drawn uniformly from offline target sequences, while at deployment each round's drafting start is set dynamically by preceding acceptance lengths, so that context-distribution mismatch remains.