A test set missing night images picks the wrong model: rewriting day photos into night restores the selection a fully covered set would make
Synopsis
When a test set underrepresents a semantic condition such as night, model selection can be misled; a generative pipeline uses vision-language models to find the missing semantics and diffusion models to turn day images into night images, so the augmented test set selects the same model as a fully covered one.
Interpretation
Missing semantic conditions in a test set can be found automatically and filled, so model selection stops being skewed by the coverage gap. Evaluators previously had only a held-out test set, with no knowledge of the training data or its semantic composition; when conditions such as night are underrepresented, the evaluation conclusions become misleading. On Caltech Camera Traps image classification and BDD100k object detection, adding synthetic night images let the test set separate a day-only model from a day-and-night model, matching the conclusions of an ideal, fully covered test set.
The pipeline runs in three steps: label images with semantics and filter for the ones that actually affect model behavior, transform existing images into the missing semantics, then verify with embedding statistics that the additions help. Semantic labels come from zero-shot binary judgments by vision-language models, meaning the image and a text description are compared by cosine similarity in a shared embedding space with no manual annotation; importance is measured by cluster recall after k-means clustering, the fraction of a semantic's samples that land in one cluster. On CCT it predicted 99 night images out of 964, with 95% precision and recall against the 100 real night images, and night had 79% cluster recall versus 60% for the next-highest semantic, winter; on BDD100k it predicted 101 night images with 98% precision and recall, and night had 81% cluster recall versus 50% for rain.
The synthetic images are used only to enrich the test set, and no downstream model is retrained. Earlier generative augmentation mostly served training; using synthetic data solely for test and evaluation, without retraining on it, had remained largely unexplored. The transformation uses CycleGAN-Turbo, a diffusion-based image-to-image model that converts day images to night while preserving scene content; embedding statistics show synthetic night images have lower drift, coverage radius and FID against real night than real day images do, and completeness is highest when all data are combined.
Perspective
The pipeline targets evaluators who have only a held-out test set and no access to training data or metadata, and it is meant to check and repair semantic coverage gaps before deployment. It has been run on binary image classification with Caltech Camera Traps and object detection with BDD100k for the day/night semantic, with additional validation on clear/rainy and clear/snowy weather semantics in BDD100k. Synthetic images enter only the test set, and downstream models are not retrained on them.
Embedding statistics confirm global semantic alignment but do not capture the fine-grained spatial features object detection needs: with mismatched synthetic data, BDD100k detection results still came close to real night images, showing the metrics have limited predictive power for dense prediction. Weather semantics are visually subtler, with lower semantic identification accuracy than day/night and less accurate importance filtering. The authors attribute the detection bottleneck to SDXL resolution and suggest larger diffusion models, while noting that strictly aligning such powerful generators with the target domain remains largely unexplored.