Quantum tests can be generated too, but running is not discriminating: best model reaches 0.71
Synopsis
Quantum programs are judged by probabilistic outputs and domain-specific checkers, so a syntactically valid test can still disagree with the benchmark verdict. QTestBench has models write syntactic, semantic, and behavioral pytest modules per task, injects candidate solutions, and compares each test verdict against the benchmark checker rather than the tests themselves across three validation cohorts. On eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, generated suites almost always execute, yet discrimination lags: the best test quality is 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality.
Interpretation
Automated test generation for quantum code now has a comparable evaluation: not whether a model writes a correct solution, but whether its tests separate correct implementations from faulty ones. Earlier quantum benchmarks such as Qiskit HumanEval, QHackBench, and QuanBench measure whether a model produces a correct quantum solution, not whether its tests distinguish correct from incorrect quantum programs. Across eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, suites from five models almost always execute, but the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval.
Test verdicts are not treated as ground truth; each one is compared against the benchmark checker, which labels every candidate correct, wrong, or unknown. Test generation and validation are split into two stages: a model first writes one pytest module per type, syntactic, semantic, and behavioral, and the framework then injects candidate solutions at runtime and runs those modules. Validation runs under three cohorts: A probes only the trusted reference to isolate executability and acceptance of known-correct code; B adds independently generated model solutions to measure discrimination against natural wrong implementations; C injects checker-verified synthetic mutants to supply calibrated negatives when natural wrong solutions are scarce.
When natural wrong solutions are scarce, synthetic mutants restore a measurable bug-detection estimate. Three of the ten Qiskit HumanEval tasks yield no wrong solutions, and there bug-detection, test-quality, and composite-quality scores are reported as not applicable rather than zero. On QHack, Configurations B and C agree closely for every model, indicating that synthetic mutants reproduce the discrimination signal of the natural wrong-solution cohort; on Qiskit, Configuration C makes bug detection defined on every task, though test-quality ceilings stay well below QHack.
Perspective
The evaluation covers eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, with tasks fixed before generation, one test-generation pass per model, and no repair or re-prompt loop. Metric definitions stay consistent across the two SDKs, and executability, acceptance, bug detection, test quality, checker-input coverage, and composite quality are aggregated under one protocol. A reader interested in where test generation sits in a quantum software workflow can use the three-cohort validation as a comparison frame, keeping in mind that single-pass generation is the current setting.
The evaluation is limited to a fixed temperature, a single generation pass without repair, ten selected Qiskit tasks, and benchmark-specific checker oracles. On Qiskit, bug detection is middling and varies widely across tasks, and three tasks cannot report bug-detection or test-quality scores because no wrong solutions appear. Configurations B and C agree closely on QHack, but the material does not establish whether that agreement holds on other task sets.