Coding agents pass 80% of PDE tasks, but about one pass in ten never solves the equation: PDEViBench traces each passing program to its computation

A pass rate on partial differential equation tasks no longer means the agent can solve the equation: on manufactured-solution tasks a program can recover the exact solution from the task data and return it, and an output check still accepts it. Earlier scientific coding benchmarks credited a run when its returned result agreed with a reference, but a manufactured task derives its forcing, boundary data and initial state from the exact solution, so agreement does not show that the equation was solved. A Codex agent (GPT-5.2) attempted each of 290 tasks five times; of 1,450 runs the endpoint pass rate was 80.1%, 8.6 percentage points (task-cluster bootstrap 95% interval 6.9–10.4) did not solve the equation, and counting only numerical solves gives a route-valid pass rate of 71.5%.

Removing shortcut passes changes what the benchmark concludes about task difficulty and about which prompt is better. Under endpoint scoring Burgers tasks looked easier than they are and the benefit of a prescribed discretisation was understated, and the two scores ranked two prompts in opposite orders. In a regression of per-task pass fraction on five design factors, counting only numerical solves lowered the operator coefficient contrasting the 40 Burgers tasks with the linear families from 0.170 to 0.109 and raised the discretisation coefficient from 0.087 to 0.141; under the prohibition prompt endpoint success fell 1.7 percentage points (95% interval 0.5–3.0) while route-valid success rose 6.9 points (5.0–8.9).

Route labels come from reading the program and, where reading cannot settle it, rerunning it on changed forcing and initial data, which a solver follows and a program tied to the original target does not. Reading alone could not decide dependency-complex programs, and two expert reviewers disagreed on 47 of 90 such programs. On 40 primary-panel programs of known route and five constructed programs the rerun test assigned every route correctly; none of the 90 dependency-complex programs returned a target-derived field, and 28 followed the change only within the original task structure and were counted as instance-specific solvers.

When the exact solution is forbidden or unavailable, passing programs are numerical solvers and stay correct on changed problem data. With the original prompt allowing the target, 125 runs took the shortcut; under the prohibition 107 of them passed by solving the equation instead. On 32 naturalistic tasks without a closed-form solution, Codex (GPT-5.5) and Claude Code (Claude Opus 4.7) each ran every task twice, all 128 programs passed and all solved numerically; of 64 replay programs 34 passed the base task and all of those passed every variant of their family.

The demonstrated scope is a 290-task PDE benchmark spanning six equation families (Poisson, Helmholtz, heat, wave, Burgers and linear elasticity) plus four two-dimensional naturalistic families; the original prompt was run with one agent, and the rerun test was applied only to programs that reading could not settle.

The share of passes that return the target rather than solve may depend on the agent; the naturalistic and replay tasks cover four two-dimensional families, leaving three-dimensional geometry, multiphysics coupling and finite-element libraries untested; and the panels change agent, model and prompt together, so the route-valid score is not yet a comparison between models.