Language models learn to revise their own drafts: decoupling generation order from display order, edits without recomputing the forward pass

A language model can revisit text it has already generated and insert, substitute, or delete an arbitrary number of tokens, instead of being locked into left-to-right generation order. Autoregressive models cannot revise their outputs after the fact, and discrete diffusion models can iteratively correct tokens but generate fixed-length output with a fixed position per token, so they cannot insert or delete arbitrary numbers of tokens. On standard benchmarks including MMLU, ARC-Challenge, GSM8K, MATH500, and HumanEval, AutoIndexer 8B LoRA essentially ties the matched Qwen3 8B LoRA baseline and consistently outperforms the LLaDA 8B and Dream 7B diffusion baselines.

The model works like a text editor: it decides when an edit should start, moves a cursor to a position in the sequence, and returns to the end to continue generating. Unlike Fill-in-the-Middle, the span to be edited is not supplied externally but predicted by the model itself. Training perturbs clean sequences with randomly sampled inverse edit operations and teaches the model to undo them; the edit diagnostic uses 100 windows per operation on the held-out validation set of the CPT data mix, with a 64-token rollout per window in free generation.

The report is for researchers and engineers working on language model decoding mechanisms. The demonstrated scope is LoRA fine-tuning on a Qwen3 8B backbone, first with continued pre-training and then with supervised instruction fine-tuning; standard benchmarks cover general, math and science, and code tasks, and the edit diagnostic covers insertion, deletion, substitution, and no-op cases. Code, the fused attention kernels, and training scripts are released.

Insertion remains the hardest correction: fewer than half of corrupted drafts triggered an edit, and when one fired it localized the insertion correctly about a quarter of the time. Under teacher forcing, the probability of emitting an edit marker at the exact ground-truth timestep is low because of an arbitrary lag between an error and its correction. After instruction fine-tuning, math, science, and code tasks drop relative to the original Qwen3 8B checkpoint, which the authors attribute to not performing reinforcement learning with verifiable rewards.