Diffusion language models without pretraining from scratch: LoRA-only conversion turns four autoregressive models diffusive in hours on one GPU
Synopsis
BYOD is a LoRA-only recipe for converting autoregressive language models into diffusion language models: every base parameter stays frozen, each layer's causal mask becomes bidirectional, and a masked-diffusion objective trains at most 102.4M tokens in 3.1–4.0 hours on one Google Colab GPU, turning Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B into models that generate from fully masked sequences, with adapters at 4.1–5.8% of final parameters. The strongest conversion, BYOD-Gemma, reaches 50.6% average accuracy on nine tasks against 51.9% for LLaDA 8B Instruct in the same harness; open-generation quality at low NFE budgets remains the main limitation.
Interpretation
An existing autoregressive model can now be turned into a diffusion language model by training only a low-rank adapter, with base weights untouched throughout. Earlier diffusion language models largely relied on costly pretraining from scratch or full-model fine-tuning; LLaDA 8B, for example, was pretrained from scratch on 2.3T tokens and 130,000 GPU-hours. One conversion per model family across four instruction-tuned families, with adapters at 4.1–5.8% of final parameters, at most 102.4M tokens per conversion, and 3.1–4.0 hours on a single Google Colab GPU.
The converted models genuinely use bidirectional context rather than keeping left-to-right generation. An autoregressive model predicts each next token from the tokens already generated to its left, and causal attention prevents right-side information from affecting earlier positions. In a right-context copy test, adding a clue placed after the masked position changed the greedy prediction on 100% of test items for every converted model; without the clue, exact-match accuracy was 0% for every model.
Conversion retains substantial parent-model knowledge, but open-generation quality still trails both the parents and LLaDA. On nine-task average accuracy, all four autoregressive parents score above their converted counterparts; BYOD-Gemma reaches 50.6% against 65.7% for its parent. Generating 128 tokens in 64 steps, BYOD perplexity is 8.3–11.1 versus 4.1 for LLaDA and 2.2–3.0 for the AR parents; BYOD-Gemma averages 50.6% accuracy against 51.9% for LLaDA 8B Instruct in the same harness.
Perspective
The demonstrated scope is conversion and paired evaluation of four instruction-tuned text models (Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, Ministral 8B Instruct 2410), evaluated on a fixed 125-example subset per benchmark, with efficiency claims about conversion training rather than measured inference speedup.
The largest gaps to the parents appear in mathematics and code, and code outputs contain syntax errors; perplexity and Distinct metrics do not measure factuality, grammar or coherence. The cross-backbone comparison is not a controlled architecture study, since the parents differ in size, tokenizer, pretraining and alignment, and fixing LoRA rank gives each backbone a different absolute adapter budget, so Gemma's lead cannot be attributed separately to architecture, parent quality, model size or trainable parameter count. Cost and carbon figures are illustrative estimates under one rental and energy scenario, not measured bills or emissions.