Pixel diffusion drops attention mixing: Volterra layers beat JiT's FID at every ImageNet scale

The mixing operator in pixel-space diffusion becomes a Volterra layer, so the residual path carries no activation function and the model uses no attention, no convolution and no pretrained components. Earlier self-contained pixel diffusion kept softmax attention, inherited from sequence modeling, as the mixing operator and changed only the prediction target to the clean image plus an aggressive per-patch bottleneck. At matched parameters and FLOPs on ImageNet, PoD has lower FID than JiT at every scale, with the largest gap at the G scale; the ordering holds at the 32×32 patch setting too, where PoD-G/32 has lower FID than JiT-G/32.

Each raw patch first passes through a low-rank linear map into a bottleneck dimension, then through stacked alternating cross-token and channel mixers that form the polynomial denoiser. The low-rank Volterra primitive replaces a dense high-order kernel with a project–multiply–mix operation, multiplying two projected groups elementwise and mixing the products back, at lower cost than a dense kernel. In a controlled one-dimensional spiral embedded in adjustable ambient dimension, the Volterra block under clean-image prediction attains the lowest mean distance to the manifold, roughly two to five times lower than the MLP and attention.

The results cover class-conditional ImageNet-1K generation at 16×16 and 32×32 patch settings and four model scales matched in parameters and FLOPs; the controlled manifold experiment uses only a one-dimensional spiral embedded in an adjustable ambient dimension.

The analysis assumes the data are supported exactly on a smooth compact manifold, whereas real image distributions only concentrate near one; the worst-case compression term can grow with the ambient dimension; and a slower rate for attention at matched budget remains a conjecture rather than a theorem.