No retraining needed for any-order image generation: one position table and an ordering curriculum unlock inpainting from old weights
Synopsis
A pretrained raster-scan autoregressive image model can only generate left to right and top to bottom, so it cannot inpaint or outpaint at inference. A new adaptation adds a future-aware positional embedding table and continues training under a curriculum that shifts from raster order to random order, letting the same weights condition on any ordering. On ImageNet, LlamaGen-XL moves from FID 2.63 to 3.91 but gains inpainting and outpainting while keeping KV caching and vLLM serving.
Interpretation
A pretrained raster-scan autoregressive image model can be turned into an any-order generator, so inpainting, outpainting and region editing become inference-time conditional generation from one set of weights rather than a separately trained model. Earlier any-order routes either train from scratch on a mixture of token orderings or switch to discrete diffusion, which by construction cannot reuse KV caching or autoregressive serving stacks such as vLLM. On ImageNet, the adapted LlamaGen-XL beats both MaskGIT schedules on inpainting and outpainting FID and has lower FID than MAR-L on both tasks, while MAR-L has higher outpainting Precision and Recall; the authors call the two competitive.
The conversion touches two things: an embedding added to each input step that names which spatial position is the next target, and a training order that moves gradually from raster scan to fully random. Without target-position information, the same visible context must explain different prediction targets and the gradients conflict; starting directly from random orderings produces large, inconsistent gradients that push the model out of the pretrained basin. Ablations show that removing the future-aware embedding keeps the loss above a substantial floor and degrades generation under all orders, and that setting the curriculum coefficient to 1 from the first step trails the full recipe on every fidelity metric for LlamaGen-L while Recall stays essentially unchanged.
Perspective
The demonstrated setting is class-conditional image generation on ImageNet with four public checkpoints, LlamaGen-L/XL and RAR-L/XL, and the editing evaluation covers region completion only. The authors note that whether the recipe extends to text-to-image autoregressive models or higher resolutions remains to be tested, and that adaptation targets token-based autoregressive models.
The authors describe the class-conditional FID increase as a measured quality-flexibility trade-off rather than a lossless conversion, and do not claim the cost is fundamental or monotonic with scale. The random-initialization control matches compute but not schedule, and the editing evaluation covers region completion only.