Worst cases stop being guesswork: Simple-PAT searches a procedural generator's parameters and cuts worst-case error on four vision tasks
Synopsis
Vision models still fail on seemingly simple scenes, and a fixed dataset cannot reliably estimate worst-case error. Procedural adversarial learning treats an interpretable procedural generator as a controllable data source and uses zeroth-order optimization to search its low-dimensional control parameters for hard instances. On four tasks spanning surface normals, image segmentation and human pose estimation, Simple-PAT lowers worst-case error below uniform training and two robust reweighting baselines against a cross-model black-box adversary with 50M aggregate queries.
Interpretation
Training data is no longer drawn from a fixed dataset; a procedural generator produces it on demand, selected against the current model. Prior adversarial training could only perturb existing images with color jitter, cropping and similar limited transformations, covering few valid image transformations, while collecting real data is slow and expensive. On the four PAL-Dojo tasks, against a cross-model black-box adversary with 50M aggregate queries, Simple-PAT reaches the lowest worst case on every task, and all sixteen comparisons give a Mann-Whitney p-value of 0.0079.
A learner and an adversary update in the same training steps: the learner runs ordinary minibatch gradient descent while the adversary searches the generator's control parameters. The adversary uses CMA-ES, a zeroth-order optimizer that adjusts its sampling distribution from the errors of sampled points without gradients through the generator. Each batch of 256 mixes 32 uniform samples with 14 samples from 16 CMA-ES instances, and the least fit 25% of instances are reset every 20 generations; the generators span 13 to 69 control parameters, and the normals task renders 8700 scenes per second.
Perspective
The demonstrated scope is the four PAL-Dojo procedural tasks: surface normal estimation, Chair-Simple and Chair-Full segmentation, and human pose estimation, with models of 5M to 7.8M parameters trained for 20,000 steps at batch size 256. The intended readers are vision researchers working on robust training and worst-case evaluation.
The true worst case cannot be computed with a finite number of queries, so the reported figure is the evaluation worst case an adversary finds within a query budget; on the 69-dimensional Chair-Full task, single-model search missed hard cases for Simple-PAT and the cross-model adversary raised its worst case, which is why the authors report cross-model evaluation as the primary metric.