Gaussian splats skip the rasterizer: GaussianRenderFormer renders directly, reaching 38.08 dB on unseen objects

Rendering a given set of 3D Gaussians no longer has to pass through a hand-written rasterizer; a transformer can output the image directly from the Gaussians and a camera. Earlier models involving Gaussians mostly reconstruct, refine, relight or analyze them, and still hand the final image to a hand-written renderer, usually the 3DGS rasterizer itself. On 300 objects never used during development, over all 28 views, rendering against the full unpruned splat reaches 38.08 dB PSNR, while rasterizing the same 20,000-Gaussian pruned input reaches 44.02 dB.

The model keeps RenderFormer's two-stage layout, encodes each Gaussian as one token, and lets a view-dependent decoder turn small patches of camera rays into pixels. The scene encoder applies a 3D rotary embedding of each Gaussian center, and the decoder adds a 2D rotary embedding of projected positions, puts each Gaussian's projected footprint on the attention keys, and shrinks image patches to a quarter of their former size. In the moderate-scale ablation, removing the footprint costs 1.7 dB, removing the 2D embedding costs 5.2 dB, and using larger patches costs 3.0 dB; at full scale the final model without the footprint scores 1.4 dB lower on the fresh held-out set.

The demonstrated setting is single objects viewed from outside at 512 pixels, a fixed field of view and camera distances from 1.15 to 2.45; the real captures are regions cut from Mip-NeRF 360 scenes and viewed the same way, not yet full scenes with the camera inside them.

The final run was still improving when it ended, gaining about 0.9 dB per doubling of steps, so under-training is not ruled out; denser inputs than in training hurt the model while helping the rasterizer, and the reachable-target experiment is a short fine-tune of an already trained model, not a from-scratch test.