A 35B verifier stops guessing verdicts: it writes its own counterexamples and lifts judging accuracy from 71% to 87%
Synopsis
A 35B agentic verifier that sees only the problem statement and a candidate program, never the official tests, uses self-generated counterexamples as executable evidence to raise binary verdict accuracy on USACO-Judge from 71.41% to 87.41%.
Interpretation
In competitive programming, a verifier can now take a problem statement and one candidate program and return both an accept-or-reject judgment and a concrete input that actually breaks the program. Official judges return a verdict under a finite submission budget without failure explanations or failing inputs, and earlier code judges could describe a suspected fault without necessarily including a replayable failing input. On USACO-Judge, the same Qwen3.6-35B-A3B backbone scores 71.41% binary verdict accuracy with the agentic scaffold and 87.41% after training, while counterexample success rises from 40.61% to 62.67%.
The verifier writes and runs its own tests inside a generic execution sandbox, then submits a verdict, a diagnosis, and a Python generator that prints one input. Training uses reinforcement learning, and most reward for a rejection requires a legal input that breaks the candidate, which blocks the shortcut of guessing the majority verdict. Training uses 20,000 candidate programs over 6,145 problems, 15% of them accepted; USACO-Judge holds 900 C++17 candidates for 39 USACO 2026 problems at a 1:2 accepted-to-unaccepted ratio.
Perspective
The reported scope is single-candidate verification: the input is a problem statement and a candidate program, the output is a verdict, a diagnosis, and a counterexample generator, and the verifier cannot see official tests, the reference solution, the input validator, or the output checker. Evaluation covers the 39 USACO 2026 problems in USACO-Judge and the hacking splits of UOJ-Bench, with training and evaluation run in a generic execution sandbox and a self-hosted online judge. It is aimed at researchers and engineers working on test-time scaling, code verification, and competitive-programming automation.
Counterexample success is only a checkable proxy for diagnosis quality: finding one establishes failure, while failing to find one does not establish acceptance. The diagnosis audit covers GPT-OSS-120B candidates only, and the authors treat it as evidence for this refinement setting rather than a cross-solver estimate. The length-penalty comparison shows that over-compression lowers mean reward and hack success from 0.777 and 62.3% to 0.755 and 59.1%, so training alternates penalized and unpenalized phases.