Changing only 5% of pixels blocks face cloning: SEAL holds identity similarity to 0.15 or less on CelebA-HQ and VGGFace2
Synopsis
Adversarial perturbations added to photos can stop a diffusion model from learning a person's identity, but earlier methods altered the whole face densely, and the accumulated distortion became visible around facial features. SEAL confines the perturbation to 5% of spatial locations, jointly optimizing where it lands and its values with a differentiable top-k mask, and disrupts the latent variance of a frozen VAE encoder. On CelebA-HQ and VGGFace2, protected images reach above 41 dB PSNR, 0.97 SSIM and at most 0.07 LPIPS, while identity similarity after DreamBooth personalization stays at most 0.15.
Interpretation
The protection perturbation no longer covers the whole image; it lands on only 5% of spatial locations, leaving the other 95% of pixels untouched. Most earlier protection methods apply bounded perturbations to every pixel, and the accumulated distortion becomes perceptible around visually sensitive structures such as the eyes, nose and mouth; recent methods improved image quality but at the cost of weaker protection. With 50 identities from each of CelebA-HQ and VGGFace2 and 4 images per identity to protect, protected images reach above 41 dB PSNR, 0.97 SSIM and at most 0.07 LPIPS, while identity similarity after DreamBooth personalization stays at most 0.15.
Where the perturbation lands is decided by optimization rather than random placement. Replacing the learned top-k mask with a random mask raises identity similarity from 0.07 to 0.68, showing that under a sparse budget the location choice carries much of the protection. The ablation runs on a 10-identity subset of CelebA-HQ with default hyperparameters, comparing the learned mask against a random mask.
The method backpropagates only through a frozen VAE encoder, needing about 70 seconds and 9 GB of GPU memory to protect one four-image identity set. Anti-DreamBooth needs about 420 seconds and 17 GB on the same task, so SEAL takes roughly one-sixth of the time and half the memory. Table 3 reports wall-clock time and peak GPU memory for protecting one four-image identity set; SEAL matches SDS(-) at 70 seconds and 9 GB.
Perspective
The demonstrated scope is 50 identities from each of CelebA-HQ and VGGFace2 with 4 images per identity to protect, a target model of Stable Diffusion v2.1 fine-tuned with DreamBooth, and evaluation under the two main prompts P1 and P2; ablations and some additional experiments use a 10-identity subset. The intended readers are researchers and engineers working on diffusion personalization and image privacy protection.
The method targets fine-tuning-based personalization, where protected images pass through the VAE encoder during training; it does not affect encoder-based personalization such as IP-Adapter FaceID. Smoothing pre-processing such as JPEG compression, Gaussian blur and resizing removes most of the sparse perturbation and largely restores identity similarity, while mild additive noise is tolerated; improving robustness to such transformations is left as future work.