Evaluation runs already leave per-question records: an audit layer reuses them and finds MMStar image zero-gain above a permutation null

The per-question predictions, judged correctness and metadata that a vision-language model evaluation already produces can now be used to audit the benchmark itself, without building a replacement benchmark or running a separate annotation campaign. Such checks were previously treated as separate research projects, either building cleaner replacement benchmarks or running dedicated flaw-mining pipelines, each with its own annotation budget and inference bill, so in practice they happened rarely and covered few benchmarks. On MMStar's 1,500 four-option items, the three-model pool shows a 20.2% zero-gain share against a 17.0% permutation null, an excess at the reportable floor of a 999-permutation test.

The audit reports every headline quantity as an excess over an item-matched permutation null, with interval estimates, leave-one-model-out ranges and category strata, so the raw share is not over-read. The raw zero-gain share itself varies by 14.2 percentage points across model pools, while the excess varies by 4.6 percentage points, showing the raw share alone is unstable. The zero-gain excess concentrates in the mathematics and science & technology categories and is absent from coarse perception, while 134 unanimous label-conflict candidates flagged by a second detector peak in the perception categories, giving the two detectors distinct category profiles.

The audit layer runs inside VLMEvalKit as six detectors plus one text-only rerun, turning saved evaluation artifacts into a per-question audit report with a single command. Mainstream vision-language model evaluation toolkits previously did not include benchmark-quality checks, which required a separate pipeline. Four of the six detectors never consult a judge and parse option letters directly; the two that do were recomputed under exact string matching, moving category shares by at most 2.1 percentage points.

The audit is validated in depth on MMStar's four-option format with three models, InternVL3-8B, Ristretto-3B and gpt-5-nano, and leave-one-model-out ranges are reported so pool sensitivity stays explicit. The audit layer, the prediction artifacts and the per-question reports are released with the submission, and the appendix gives the commands that regenerate every number and figure. The binary-format boundary is measured and reported: on HallusionBench the detectors do not recover the benchmark's own design labels.

A pool of three models fixes the direction of both signals, but the exact category shares remain pool-dependent, which the authors treat as a working approximation whose sensitivity is measured. The zero-gain excess is an aggregate, mechanism-agnostic quantity that bounds the combined effect of language priors, option statistics and contamination, and the category strata localize it without singling out one cause. The 134 unanimous label-conflict questions stay candidates until human review turns them into a measured error rate, with unanimity serving as a prioritization signal.