LLM-written quantum tests almost always run: best generator scores 0.71 test quality on QHack but only 0.34 on Qiskit
Lead
Large language models write quantum tests that almost always execute, yet their ability to separate correct from faulty implementations lags well behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality.
Story
The quality of tests for quantum code can now be scored on its own, rather than only asking whether a model writes a correct solution. QTestBench has models write pytest test modules per task, injects candidate solutions at runtime, and compares each test verdict with the label assigned by the benchmark's own checker. Earlier quantum benchmarks such as Qiskit HumanEval, QHackBench and QuanBench measure whether a model produces a correct quantum solution, not whether its tests can separate correct implementations from faulty ones. Across eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, generated suites from five models almost always execute, but the best generator reaches a test quality of only 0.71 on QHack and 0.34 on Qiskit HumanEval.
Tests are generated in three kinds: syntactic tests check structure and entry points, semantic tests execute the candidate on official inputs with numerical comparison, and behavioral tests run the end-to-end harness path. The three kinds rest on different ground truth: syntactic tests never execute a circuit, semantic tests compare against embedded official input/output values, and behavioral tests call the candidate's own run()/check() helpers on QHack or invoke the injected entry point on Qiskit. Validation runs under three cohorts: Configuration A probes only the trusted reference to isolate executability and acceptance of known-correct code, Configuration B adds independently generated model solutions as natural wrong probes, and Configuration C injects synthetic mutants that the checker confirms are wrong when natural wrong solutions are scarce.
When natural wrong solutions are too sparse, synthetic mutants make bug detection measurable again. Three of the ten Qiskit tasks attract no wrong solutions under Configuration B, and there the bug-detection, test-quality and composite-quality scores are reported as not applicable rather than as zero. Configuration C applies deterministic mutations to the trusted reference and keeps only mutants the official checker labels wrong, trying up to twelve candidates and keeping the first three wrong probes, so bug detection is defined for every task.
What to watch
The evaluation covers eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, fixed before generation as an easy, intermediate and hard mix, with five models each generating tests in one pass and no repair or re-prompt loop. It is aimed at researchers working on quantum code generation and software-test trustworthiness, and at engineers who need to judge whether generated tests can guard quantum code. The material names refinement with execution feedback as a natural next step.
The evaluation is limited to a fixed temperature, a single generation pass without repair, ten selected Qiskit tasks, and each benchmark's own checker as ground truth. Qiskit scores are lower and more heterogeneous across tasks, partly because many checkers contain few extractable literals, as in the Deutsch–Jozsa task. Absolute test-quality scores depend on probe composition, so bug detection should be read together with probe counts and is reported as not applicable when a cohort contains no wrong probes.