GRAFT swaps trajectories between two heterogeneous models, beating GRPO for both at equal budget with a 2.1-point average gain
The work proposes GRAFT, a framework that in Reinforcement Learning with Verifiable Rewards (RLVR) replaces a receiver's all-fail rollout groups with peer groups containing both successful and unsuccessful responses, controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping; across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT improves both models over GRPO at the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points, with 1.8 points on average preserved when reusing stored peer trajectories.