First systematic empirical study of diffusion LLMs for code generation: 7 diffusion models reach 79.1% on MBPP+ versus 73.3% for the best autoregressive baseline, yet open-source diffusion models still do not consistently match strong autoregressive models
This study empirically evaluates 7 representative diffusion LLMs (including the closed-source Mercury-Coder-Small) against 4 open-source autoregressive baselines on HumanEval/HumanEval+, MBPP/MBPP+, HumanEval-X, LiveCodeBench, RepoQA, RepoBench-C, and SWE-Bench Verified, finding promising but uneven code-generation ability: the closed-source diffusion model leads on most benchmarks (e.g., 79.1% on MBPP+ versus 73.