Offline demos and fresh rollouts now share one regression target: unified training lifts math-reasoning avg@32 by 2.33 points

Reasoning post-training no longer needs a supervised fine-tuning stage followed by a switch to reinforcement learning; offline demonstrations, current-policy rollouts, and historical rollouts can enter one reward-labeled replay buffer and be trained with a single regression loss throughout. The conventional pipeline first runs cold-start supervised fine-tuning on positive offline chain-of-thought demonstrations, then switches to reinforcement learning from verifiable rewards on model-generated rollouts; the two stages use different objectives, negative demonstrations are excluded from fine-tuning, positive demonstrations are retired once reinforcement learning begins, and earlier rollout batches are replaced even when their verifier rewards remain informative. On Qwen3-1.7B, the unified configuration reaches benchmark-averaged avg@32 and pass@32 of 44.57 and 68.46, which is 2.33 and 4.44 percentage points above the 42.24 and 64.02 of two-stage SFT plus DAPO, and it leads on all eight benchmark-metric entries.

Each completion gets one regression target built from its own verifier reward, reference-policy log-probability, and a frozen prompt-level normalizer, so positive and negative completions from different policies share the same per-sample loss. That target does not contain the behavior policy that generated the completion, so training needs no stored behavior-policy probabilities, no importance weighting, and no learned critic; offline, on-policy, off-policy, and unified settings are just restrictions on which data sources are trainable. Training uses 16,254 decontaminated math prompts, each paired with two positive and one negative DeepSeek-R1-Distill-Qwen-32B chain-of-thought trajectories; the core configuration generates 650,160 responses over 1,280 optimizer steps and is evaluated on 842 prompts from AIME25, MinervaMath, AMC23, and MATH500 with 32 responses each.

Experiments cover Qwen3-1.7B and Qwen3-8B, are limited to mathematical reasoning with rule-based verifiers, and evaluate on 842 prompts from AIME25, MinervaMath, AMC23, and MATH500 with 32 responses per prompt. The intended readers are researchers and engineers working on reasoning post-training and reinforcement-learning data reuse. The source states no next-step experiment.

The source states that experiments are limited to mathematical reasoning with Qwen3 models and rule-based verifiers, leaving generalization to other domains, model families, and noisy or subjective reward signals untested; the frozen prompt normalizer also incurs an upfront generation cost, and behavior under offline distribution shifts across domains or model families remains untested.