# Reinforcement Learning with Verifiable Rewards

Dory1.0-4B is a 4-billion-parameter looped language model whose single checkpoint varies how many times a shared recurrent block runs at inference, letting one model serve different compute budgets. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, then extended to a 256K context and post-trained with SFT and RLVR. On the equal-weight average of 12 benchmarks it rises from 35.2 at one loop to 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.

Dory1.0-4B is a 4-billion-parameter looped language model whose single checkpoint varies how many times a shared recurrent block runs at inference, letting one model serve different compute budgets. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, then extended to a 256K context and post-trained with SFT and RLVR. On the equal-weight average of 12 benchmarks it rises from 35.2 at one loop to 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.

GRPO scores every rollout of a prompt against one shared group-mean baseline, yet rollouts of the same prompt often follow different solution strategies with different reward distributions. This work defines the subgroup offset as a subgroup mean's deviation from the global mean and shifts the baseline along it by a single signed coefficient: the positive sign pulls credit toward a rollout's standing inside its subgroup, the negative sign pulls it toward the subgroup's overall quality. On Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, both directions give modest but consistent gains over an anchor that shares the same strategy-conditioned sampling and differs only in the baseline, concentrated in pass@16, with the negative direction stronger.

GRPO scores every rollout of a prompt against one shared group-mean baseline, yet rollouts of the same prompt often follow different solution strategies with different reward distributions. This work defines the subgroup offset as a subgroup mean's deviation from the global mean and shifts the baseline along it by a single signed coefficient: the positive sign pulls credit toward a rollout's standing inside its subgroup, the negative sign pulls it toward the subgroup's overall quality. On Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, both directions give modest but consistent gains over an anchor that shares the same strategy-conditioned sampling and differs only in the baseline, concentrated in pass@16, with the negative direction stronger.

On-policy distillation trains a student on trajectories the student itself generates, but student prefixes can carry reasoning errors or patterns unfamiliar to the teacher, and the teacher's token-level supervision becomes less reliable as prefixes grow longer. Before distillation begins, SPTA trains the teacher on student prefixes of random lengths, rewarding correct final answers, so the teacher learns to recover from student-induced states rather than imitate the student's reasoning; standard on-policy distillation then proceeds with the distillation objective unchanged. On mathematical reasoning benchmarks, students trained with SPTA average better than under standard on-policy distillation and better than under on-policy distillation with task-level RLVR-adapted teachers.

On-policy distillation trains a student on trajectories the student itself generates, but student prefixes can carry reasoning errors or patterns unfamiliar to the teacher, and the teacher's token-level supervision becomes less reliable as prefixes grow longer. Before distillation begins, SPTA trains the teacher on student prefixes of random lengths, rewarding correct final answers, so the teacher learns to recover from student-induced states rather than imitate the student's reasoning; standard on-policy distillation then proceeds with the distillation objective unchanged. On mathematical reasoning benchmarks, students trained with SPTA average better than under standard on-policy distillation and better than under on-policy distillation with task-level RLVR-adapted teachers.

Training data for multimodal perception no longer has to be built task by task. Spatial perception and long-context understanding previously relied on costly annotation, so EVIDENCECONNECT instead picks 3 to 7 related regions in one image and synthesizes multiple-choice questions that require comparing, aggregating or chaining evidence from zoomed views, then trains base models with RLVR. ZwZ2-4B rises from 37.43 to 45.26 on ViewSpatial-Bench and from 60.08 to 70.08 on MileBench, with its 25-benchmark average moving from 49.41 to 54.79.

Training data for multimodal perception no longer has to be built task by task. Spatial perception and long-context understanding previously relied on costly annotation, so EVIDENCECONNECT instead picks 3 to 7 related regions in one image and synthesizes multiple-choice questions that require comparing, aggregating or chaining evidence from zoomed views, then trains base models with RLVR. ZwZ2-4B rises from 37.43 to 45.26 on ViewSpatial-Bench and from 60.08 to 70.08 on MileBench, with its 25-benchmark average moving from 49.41 to 54.79.

Verifiable instruction-following benchmarks check whether output constraints are met, not whether the response answers the request, so frontier models can earn full credit with little substantive content. Using the same verifier as a reinforcement-learning reward produces reward hacking: publicly released OLMo 3 checkpoints show higher constraint satisfaction but lower response quality. CaLIF uses two likelihoods from the frozen reference policy, one under the request and one under the request plus constraints, to define a band that pays a fixed bonus inside and zero above it, raising IFBench constraint satisfaction from 26.98 to 34.64 across five configurations from 1.7B to 8B while holding response quality at 4.59.

Verifiable instruction-following benchmarks check whether output constraints are met, not whether the response answers the request, so frontier models can earn full credit with little substantive content. Using the same verifier as a reinforcement-learning reward produces reward hacking: publicly released OLMo 3 checkpoints show higher constraint satisfaction but lower response quality. CaLIF uses two likelihoods from the frozen reference policy, one under the request and one under the request plus constraints, to define a band that pays a fixed bonus inside and zero above it, raising IFBench constraint satisfaction from 26.98 to 34.64 across five configurations from 1.7B to 8B while holding response quality at 4.59.