# Adversarial Perturbation

Adversarial perturbations added to photos can stop a diffusion model from learning a person's identity, but earlier methods altered the whole face densely, and the accumulated distortion became visible around facial features. SEAL confines the perturbation to 5% of spatial locations, jointly optimizing where it lands and its values with a differentiable top-k mask, and disrupts the latent variance of a frozen VAE encoder. On CelebA-HQ and VGGFace2, protected images reach above 41 dB PSNR, 0.97 SSIM and at most 0.07 LPIPS, while identity similarity after DreamBooth personalization stays at most 0.15.

Adversarial perturbations added to photos can stop a diffusion model from learning a person's identity, but earlier methods altered the whole face densely, and the accumulated distortion became visible around facial features. SEAL confines the perturbation to 5% of spatial locations, jointly optimizing where it lands and its values with a differentiable top-k mask, and disrupts the latent variance of a frozen VAE encoder. On CelebA-HQ and VGGFace2, protected images reach above 41 dB PSNR, 0.97 SSIM and at most 0.07 LPIPS, while identity similarity after DreamBooth personalization stays at most 0.15.

Using a controlled two-stage PGD attack on 200 held-out COCO images with Qwen2.5-VL-7B-Instruct, the authors show that a targeted perturbation can drive teacher-forced training loss for a fixed target caption to near zero while free generation still produces the original correct description, localize this train/inference gap to a single autoregressive step (target-token rank fixed at exactly 3,488 of 152,064 vocabulary entries with zero variance), and trace all 28 decoder layers to show the visual encoder corrupts every image by a comparable margin while the language decoder amplifies the corrupted signal for susceptible images and actively suppresses it for resistant ones (p<0.001, rank-biserial r=0.579).

Using a controlled two-stage PGD attack on 200 held-out COCO images with Qwen2.5-VL-7B-Instruct, the authors show that a targeted perturbation can drive teacher-forced training loss for a fixed target caption to near zero while free generation still produces the original correct description, localize this train/inference gap to a single autoregressive step (target-token rank fixed at exactly 3,488 of 152,064 vocabulary entries with zero variance), and trace all 28 decoder layers to show the visual encoder corrupts every image by a comparable margin while the language decoder amplifies the corrupted signal for susceptible images and actively suppresses it for resistant ones (p<0.001, rank-biserial r=0.579).

Under a white-box threat model, the work aligns each stream of a source image with its counterpart in a target image in the victim vision-language model's post-merger token space and evaluates under a strict success criterion requiring the model to simultaneously name the target, confirm its presence, and deny the source, finding that target semantics appear at ε=2/255 and complete replacement reaches 38% at ε=4/255 on images and 35.9% at ε=1/255 on video, while also observing semantic fusion.

Under a white-box threat model, the work aligns each stream of a source image with its counterpart in a target image in the victim vision-language model's post-merger token space and evaluates under a strict success criterion requiring the model to simultaneously name the target, confirm its presence, and deny the source, finding that target semantics appear at ε=2/255 and complete replacement reaches 38% at ε=4/255 on images and 35.9% at ε=1/255 on video, while also observing semantic fusion.