PixVid moves video generation back to pixels: one token per frame, camera control lands on the next frame
Synopsis
PixVid converts a pretrained latent video diffusion model into a frame-wise pixel-space generator: it keeps the diffusion transformer, replaces the VAE decoder with a convolutional detailer head, and finetunes from Wan2.1-14B. It reaches VBench totals of 85.09 at 960p and 84.78 at 480p, above Wan2.1's 84.05 and PixelUMM's 83.72, and keeps 98% of source motion on fast clips versus 83% for an identically finetuned latent model.
Interpretation
Video generation no longer has to pass through VAE spatial and temporal compression: in PixVid every token covers a patch of exactly one frame, so conditions, actions and edits attach to individual frames. Earlier latent video models used a 3D VAE to merge several frames into one latent that corresponds to no single frame, so conditions applied to groups of frames, objects deformed and edits misaligned under fast motion, and autoregressive generation had to wait for a full block of controls. On VBench, PixVid scores 84.78 total at 480p and 85.09 at 960p, against 84.05 for Wan2.1 and 83.72 for the from-scratch pixel model PixelUMM; in a human study raters preferred PixVid across all comparisons.
The conversion changes only the output interface: the pretrained DiT is kept, the input pixel patch embedding is initialized from a linear map fitted to the VAE encoder, and a lightweight causal convolutional detailer head renders tokens into pixels. Training a pixel-space video model from scratch demands prohibitive data and compute, while a pretrained latent model already captures high-level semantics and dynamics. Finetuned from Wan2.1-14B, it converges within M training clips and transfers across Wan2.1 and Wan2.2-based models with different VAEs; ablations show the convolutional head lowers median patch seams from 118.6 to 13.3 and clean-video prediction lowers median pixel noise from 62.25 to 0.78.
Frame-wise pixel generation eases three structural limits at once: resolution, control alignment under fast motion, and control-to-screen latency. Latent models gain more tokens and attention compute when resolution doubles, and temporal compression delays every control by at least one frame block. Doubling the patch from 8 to 16 lets PixVid generate 960p with the same per-frame token count as 480p; depth-conditioned editing on fast clips gives motion error 10.7 pixels and depth error 0.109 versus 17.9 and 0.136 for the identically finetuned latent model; after Self Forcing distillation, median camera-control latency falls from 495 ms to 34 ms.
Perspective
The demonstrated scope covers text-to-video, 960p high-resolution generation, video editing from per-frame depth, and camera-controlled streaming autoregressive generation; evaluation uses VBench, a human preference study on Movie Gen Video Bench prompts, stock videos with Video Depth Anything depth, and camera paths from nine public datasets.
Without temporal compression, PixVid processes as many tokens per second of video as Wan2.1 at 480p, so short low-resolution clips cost more; frame-wise generation has lower throughput than SolarWM (29.4 versus 72.0 fps) and higher camera rotation and translation errors (5.44° versus 4.24°, 0.312 versus 0.231), which the authors attribute to drift over a longer pose sequence and leave to future work; VBench resizes frames to at most pixels before scoring, so it does not measure the added detail at 960p.