HDL locates branch points by hindsight divergence, cutting generated tokens 35–61% and lifting agent tasks by up to 12.46 points

HDL turns "where the model changes its mind after learning the outcome" into a computable score: given a completed root trajectory and verifier feedback, the policy first generates a reflection, then re-scores the sampled tokens under the original context and under a hindsight context containing the feedback and reflection, using the absolute difference in log-likelihood as the hindsight-divergence score and selecting the highest-scoring positions as branch points. Earlier branching methods choose positions by policy entropy (TreeRL, BPO) or by asking the model to name retry points (PivoARL, R3L); HDL ranks positions by hindsight-induced likelihood change and uses hindsight only for branch selection, not as a supervision signal. The paper gives the scoring formulation and a code-generation example: the root's primality test wrongly accepts a number, feedback prompts a reflection identifying the missing guard, and the largest log-likelihood change falls on the for position; on Qwen3-8B, HDL scores highest in all three domains against the Entropy and Reflection localization signals, exceeding them by 5.67 and 6.54 percentage points on agent tasks.

A training group is formed from a few complete roots plus branch continuations: each continuation reuses its root prefix and samples a fresh suffix under the original task context, and the loss applies only to newly generated tokens, lowering generation cost without changing group size or the GRPO objective. Asynchronous RL systems and partial rollouts improve throughput but keep the complete-trajectory sampling unit; token-selective methods choose tokens for updates only after complete trajectories are generated and do not reduce generation. HDL moves budget to branch continuations at the generation stage. The default configuration samples 2 roots per problem, selects the two highest-scoring positions per root, and allocates 3 and 4 continuations to them, giving 12 trajectories per group; generated tokens fall 35–61% versus GRPO and 43–75% versus DAPO, and rollout wall-clock time falls 18–45% versus GRPO and 34–69% versus DAPO, with token counts already including HDL's reflections and DAPO's discarded candidates.

At matched group sizes and training steps, HDL outperforms GRPO in math, code, and agent tasks, with the largest gains on agent tasks. The result indicates that concentrating exploration on decisions the model reconsiders in hindsight can improve performance under a smaller generation budget, not merely save compute. Three models (Qwen3-4B, Qwen3-8B, Llama3.1-8B) are evaluated on AIME24/25/26, HMMT, Minerva, OlympiadBench, LiveCodeBench v5/v6, and ScienceWorld, with evaluation every 10 steps and the three highest-scoring checkpoints per domain reported; Qwen3-8B math averages 54.16% versus GRPO 53.17% and DAPO 53.47%, code reaches 54.15% (Qwen3-4B) and 55.56% (Qwen3-8B), and agent scores rise from 57.76% to 67.44% (Qwen3-4B) and from 59.50% to 71.96% (Qwen3-8B).

Branching configuration and localization signal involve measurable trade-offs: the default 2 roots by 2 points scores highest on Qwen3-8B agent (71.96%), reducing to one branch point per root saves a further 15% of rollout time at a 0.87-point score drop, and increasing to four roots generates more tokens and scores 4.09 points lower. This turns "where and how much to branch" from design intuition into a measurable configuration choice, favoring revisiting multiple positions per root while keeping several continuations per position. Three configurations are compared at the same group size with reported score, generated tokens, and rollout time; the Reflection baseline yields an average of 3.22 valid branch points out of a maximum of four, with some returned positions unparseable or too close together.

The method targets settings with verifiable rewards and usable feedback. The paper validates it on three domains: math (4,555 problems after filtering DeepMath-103K), code (4,063 problems after combining and filtering DeepCoder and rStar-Coder subsets), and agent tasks (ScienceWorld, 1,856 task–variation pairs, episodes limited to 30 actions), with Qwen3-4B, Qwen3-8B, and Llama-3.1-Nemotron-Nano-8B-v1, trained with the slime framework on four nodes of four GB200 GPUs each. It suits teams that want to compress rollout generation cost without changing group size or the GRPO objective, particularly for long-horizon interactive tasks; the next question is whether hindsight divergence remains an effective selection criterion at larger model scales, in more verifiable domains, and under different feedback granularities.

The body tables in the evidence bundle (per-domain math, code, and agent scores, the localization-signal comparison, and the branching-configuration comparison) are empty, so only the aggregate numbers stated in the prose can be cited and individual benchmark scores and standard deviations cannot be checked. The appendix notes that HDL depends on the policy reliably interpreting feedback: outcome agreement with the verifier is 99.7–99.9% for Qwen3-8B but drops to 76.0% on math and 56.1% on code for Qwen3-1.7B, whose agent performance matches GRPO (45.70% vs 45.66%) and trails Entropy (47.54%). Whether hindsight re-scoring introduces noise in weaker models, and how reflection generation and hindsight scoring overhead scale, remain open questions worth watching.