Power sampling fails to converge at real compute budgets, and simple Best-of-N matches its reasoning gains
Synopsis
MCMC and SMC samplers for the power distribution fail to converge under realistic compute budgets, and their reasoning gains are largely recreated by simple Best-of-N selection; Power-RLOO, which fine-tunes directly toward the power distribution, helps only when the base model is already aligned with the task.
Interpretation
Power-distribution sampling, recently used to sharpen LLM reasoning at test time, does not actually converge to its target under realistic compute budgets. Earlier work attributed the reasoning gains of MCMC and SMC samplers to faithful approximation of the power distribution. On MATH500 and GPQA Diamond, Power-SMC and selection baselines such as Best-of-N with Resampling reach similar accuracy, indicating the gains come from picking high-probability sequences rather than approximating the power distribution.
Power-RLOO instead fine-tunes the base model to approximate the power distribution directly, sidestepping sampler convergence. Previously the power distribution could only be approximated by MCMC or SMC sampling. The method casts the power distribution as a KL-regularized RL objective with log-likelihood rewards, and a LoRA adapter learns only the lookahead correction beyond low-temperature sampling, trained on Qwen2.5-7B and Phi-3.5-mini-Instruct.
Faithful approximation of the power distribution yields accuracy gains that are noisy and depend on the model and task. Approximation quality itself was assumed to drive reasoning improvements. On Qwen2.5-7B-Instruct, Power-RLOO reaches 78.4% on MATH500 and 41.0% on GPQA, but on the weaker Phi-3.5-mini-Instruct it scores 43.6% on MATH500, below Power-SMC's 46.6%.
Perspective
The report covers the test-time sharpening setting: MCMC, SMC, Best-of-N and Power-RLOO are compared on MATH500 and GPQA Diamond. It is intended for researchers and engineers working on reward-free reasoning and test-time compute allocation. The source states no next-step experiment.
The source reports that Power-RLOO's gains are noisy and exceed low-temperature decoding only when the base model is already aligned with the task; on Qwen-Base it collapsed to overly short completions, while SMC and Best-of-N avoid this precisely because they fail to converge. Mild sharpening improves pass@k, but Power-SMC's pass@k suffers from particle degeneracy.