Chroma keying drops the aligned plate: one to three unaligned background images drive joint foreground and background generation

When compositing a performance video, the background references need not match the foreground in camera pose, crop or layout; the model infers where the foreground belongs in the referenced environment and how the background evolves with foreground motion and implied camera behavior. Earlier generative compositing either required pre-aligned foreground and background videos and mainly relit an independent foreground layer, or constrained the environment through an edited first frame or keyframes, leaving newly revealed regions prone to drift. On 500 foreground videos paired with two different sets of unaligned background references, giving 1,000 test cases, FreeCHROMA scores highest on all four vision-language criteria: foreground preservation, edge quality, foreground-background fusion at 7.728 and illumination harmony at 7.956.

The paired supervision for training comes from ordinary monocular videos: the original video is the target, its foreground is separated and its illumination perturbed, and unaligned background references are generated from selected keyframes, so the model learns to reconstruct the original video from those inputs. Supervising directly on the original video lets the model learn to copy the foreground illumination, because inside the matte the conditioning input and the regression target are identical; perturbing only the conditioning path while leaving the target unchanged makes copying no longer optimal. Removing foreground illumination perturbation drops foreground-background fusion from 7.728 to 6.384 and illumination harmony from 7.956 to 6.958; removing background viewpoint augmentation raises absolute trajectory error from 0.004 to 0.005 and lowers relative path length from 0.940 to 0.909.

The implementation trains only a small set of added parameters: the foreground and mask input-projection weights, the background-reference convolution and LoRA adapters, while the backbone video diffusion weights, VAE and text encoder stay frozen. Inference uses no auxiliary foreground relighting network, per-frame harmonizer, explicit depth loss or reconstruction model, because background synthesis and video-level harmonization happen within one generation. The model is initialized from Wan2.2-Fun-A14B-Control and trained on eight NVIDIA H200 GPUs in BF16 on 81-frame, 720x1280 clips, taking each expert's checkpoint at step 4,500, with 25 sampling steps and a classifier-free guidance scale of 4.5 at inference.

The method targets short clips: one to three unaligned references supply clip-level context, the foreground mask decides which elements and motions are retained, and the rest of the scene is generated. Evaluation covers 500 foreground videos with two background reference sets each, 1,000 cases drawn from HumanVid-Real and OpenVidHD and spanning static to large camera motion. The robotics direction is exploratory on selected RoboCasa and DROID episodes without robot-specific training.

The references provide no persistent scene memory for long-form generation, and independently generated shots lack precise cross-shot foreground-background consistency beyond coarse textual guidance. The output is not unique, because the references are not target frames and no pixel correspondence, camera calibration or pose alignment is assumed; the objective is conditional plausibility and reference fidelity under foreground preservation. The robotics experiments include both successful and failed outcomes.