# GRAFT

Continuous diffusion language models generate a whole sentence in parallel without drawing a word at intermediate steps, and the cost is a collapsed lexical tail: ELF nearly matches human text on common words, yet its rarest words appear at only a fraction of the human rate, a gap that persists from 105M to 652M parameters and under every sampler setting tested. GRAFT reads the decoding head mid-generation, commits the rare words the average is about to discard, and regenerates from noise, raising the rare-word share from 0.42–0.77 to 0.82–1.25 of the human rate across three ELF scales and the independent LangFlow implementation, at the cost of a small rise in non-word rate on some models.

Continuous diffusion language models generate a whole sentence in parallel without drawing a word at intermediate steps, and the cost is a collapsed lexical tail: ELF nearly matches human text on common words, yet its rarest words appear at only a fraction of the human rate, a gap that persists from 105M to 652M parameters and under every sampler setting tested. GRAFT reads the decoding head mid-generation, commits the rare words the average is about to discard, and regenerates from noise, raising the rare-word share from 0.42–0.77 to 0.82–1.25 of the human rate across three ELF scales and the independent LangFlow implementation, at the cost of a small rise in non-word rate on some models.

GRAFT introduces a continual multi-teacher knowledge distillation framework in which the previously distilled model acts as a teacher to preserve learned capabilities while the current student jointly learns from that model and the incoming teacher, and it adds Teacher Specific Readout Tokens plus a Geometry Agnostic Relational Loss to reconcile the incompatible representation geometries of heterogeneous teachers, yielding a single continually extensible backbone that unifies image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language, with each new capability acquired at the cost of a single distillation rather than a full re-distillation.

GRAFT introduces a continual multi-teacher knowledge distillation framework in which the previously distilled model acts as a teacher to preserve learned capabilities while the current student jointly learns from that model and the incoming teacher, and it adds Teacher Specific Readout Tokens plus a Geometry Agnostic Relational Loss to reconcile the incompatible representation geometries of heterogeneous teachers, yielding a single continually extensible backbone that unifies image understanding, 2D dense prediction, 3D human pose estimation, 3D vision, and vision-language, with each new capability acquired at the cost of a single distillation rather than a full re-distillation.

The work proposes GRAFT, a framework that in Reinforcement Learning with Verifiable Rewards (RLVR) replaces a receiver's all-fail rollout groups with peer groups containing both successful and unsuccessful responses, controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping; across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT improves both models over GRPO at the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points, with 1.8 points on average preserved when reusing stored peer trajectories.

The work proposes GRAFT, a framework that in Reinforcement Learning with Verifiable Rewards (RLVR) replaces a receiver's all-fail rollout groups with peer groups containing both successful and unsuccessful responses, controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping; across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT improves both models over GRPO at the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points, with 1.8 points on average preserved when reusing stored peer trajectories.