QuSema uses quantum semantics as a source-level oracle to relocate 20 historical silent bugs and surface 40 maintainer-confirmed new bugs in Qiskit and PennyLane

The paper characterizes silent bugs in quantum libraries as semantic deviations that produce invalid outputs from valid inputs without an explicit failure signal, and uses data-flow analysis to localize each deviation to the first library API that turns a valid input into an invalid output. Prior differential and metamorphic testing rely on black-box oracles, namely cross-library comparable executions or semantics-preserving circuit transformations; this work moves the oracle to source-level implementation logic and documentation, so it does not require predefined relations between execution results. The authors screened 3,184 bug reports from Qiskit and PennyLane using Codex with GPT-5.6-Sol (Ultra), searching for cases where each API behaves correctly in isolation but a permitted composition fails; only two candidates surfaced, Qiskit #4369 and PennyLane #534, and both produce an explicit exception rather than a silent incorrect result, so no counterexample was found.

QuSema builds contextual units per code segment, combining the target segment with documentation that constrains its intended behavior and call relations that give access to related implementation logic. Compared with analyzing a whole source file or an isolated function, this design compresses unrelated context while retaining dependencies; the paper notes that the Qiskit 2.4.1 Rust file standard_gates_commutations.rs contains 7,778 lines covering gate metadata, commutation tables, and helper logic. In the ablation, the base configuration analyzing complete source files identified 12–15 defects across three runs (mean 13.67); adding Contextual Unit Preparation raised the mean to 15.00 with 15 defects in each run, and adding the source-level semantic oracle raised it to 15.67.

On a benchmark of 20 maintainer-confirmed historical silent bugs, QuSema relocates more bugs than the compared general coding agents, and with DeepSeek as its backend it does so at substantially lower cost. The benchmark holds 10 bugs per library across 10 functionality types; 10 bugs lack comparable implementations in the other library, and for 8 more, equivalent circuit variants can be constructed but their execution results do not expose the defects, placing these cases beyond the direct coverage of differential and metamorphic testing. Across three independent runs, QuSema with Opus 5 averaged 16.00 relocations (USD 1,603.44), QuSema with DeepSeek 15.67 (USD 162.27), Claude Code with Fable 5 14.67 (USD 425.04), and Codex with GPT-5.6-Sol 13.33 (USD 117.29); each run produced 56–73 candidates, of which 54–72 remained after validation.

On Qiskit 2.4.1 and PennyLane 0.45.0, QuSema reported 40 previously unknown bugs confirmed by maintainers, of which 30 are silent bugs and 10 cause crashes or exceptions on valid inputs. The findings span circuit transformation, symbolic computation, commutativity, measurement, resource estimation, serialization, and execution; 5 of the silent bugs affect PennyLane functionalities that have no directly comparable API in the evaluated Qiskit SDK and no applicable circuit-based metamorphic relation. QuSema analyzed 68,687 lines and 2,753 code segments in Qiskit and 72,262 lines and 2,269 code segments in PennyLane; maintainer feedback mentions a pushed fix in #16428 and asks whether the authors would submit PRs for the issues they are discovering, noting that QuSema already pinpoints the exact root cause for each problem.

The work targets API-local silent bugs, where the semantic deviation can be localized to the first library API that transforms a valid input into an invalid output; the paper explicitly does not directly address bugs arising solely from interactions among individually correct APIs. The intended audience is testing researchers and library maintainers who need to audit quantum library implementation logic, and the intended setting is a library with parseable source, associable documentation, and executable APIs, as validated on Qiskit 2.4.1 and PennyLane 0.45.0. The method relies on LLM knowledge of quantum computing and software and uses documentation as semantic evidence of intended behavior; the paper notes that whether the documentation itself is correct is not independently established, so documentation errors may affect bug assessment and require maintainer review. For closed-source or poorly documented libraries, or different implementation languages and binding systems, applying the approach may require changes to code segmentation, call-relation analysis, and documentation association.

A careful reader would still watch several things: LLM outputs vary across runs and model versions, and although the paper repeats each configuration three times and reviews candidate reports manually, assessment may still involve subjective judgment; RQ1 compares QuSema with Claude Code and Codex as complete configurations, so performance differences cannot be attributed to agent design alone; public benchmark issues and fixes may have appeared in LLM training data despite being withheld from the evaluated methods; RQ3 reduces this threat by evaluating previously unknown bugs, but prior exposure to the underlying library source code cannot be excluded. In addition, 10 of the 40 new bugs are crashes or exceptions rather than silent bugs, which is worth noting when reading against the silent-bug detection goal; the paper does not report validation on cross-library, cross-language, or closed-source settings, which remain open questions.