LUCA BLUNDA

Discrete diffusion models have lacked the statistical theory that continuous diffusion enjoys, and it was unclear whether they can exploit sparse structure in the data distribution. This analysis casts masked-diffusion score estimation as a per-context label-smoothed classifier and shows the smoothing is equivalent to regularizing with the score loss of the uniform distribution. On sparse distributions where at most k sequences have positive probability, the terminal laws of both masked and uniform noising attain the sparse minimax rate, and the smoothing level can be chosen adaptively from the number of distinct training sequences without knowing the sparsity parameter.

Discrete diffusion models have lacked the statistical theory that continuous diffusion enjoys, and it was unclear whether they can exploit sparse structure in the data distribution. This analysis casts masked-diffusion score estimation as a per-context label-smoothed classifier and shows the smoothing is equivalent to regularizing with the score loss of the uniform distribution. On sparse distributions where at most k sequences have positive probability, the terminal laws of both masked and uniform noising attain the sparse minimax rate, and the smoothing level can be chosen adaptively from the number of distinct training sequences without knowing the sparsity parameter.