Coding agents' rule files stop being added by impression: four runs lift held-out pass rates from 22% to 77%

The text files that steer a coding agent are no longer added or removed by impression: a new skill or rule is kept only when repeated comparisons against the current harness show a net gain and no task gets noticeably worse, and deleting existing text goes through the same comparison. Such text was previously judged by watching it work once, but the same agent on the same task can pass one attempt and fail the next, so a single success proves little. Across four independent runs, the evolved harness beat a harness with no skills or rules by 55.0 percentage points on 23 convention-family scenarios no session had played, raising the pass rate from 21.5% to 76.5%, with a 95% confidence interval of [+34.1, +74.1]; the control family of plain bugs moved by −0.8 points and no run kept an item for it.

It turns working sessions themselves into evidence: a fix that took a session from broken to working is saved as a replayable task, each correction the developer makes is logged as a one-line lesson, and the loop writes one new skill or rule at a time from those lessons. A new item first runs a cheap screen on the currently failing tasks, and only candidates that survive it pay for a stricter confirm over every valid task, with the verdict computed by a script against fixed gates rather than by a model. The four runs recorded 39 cycles, 14 of which proposed nothing because no recurring problem cleared the bar; the other 25 swept 26 candidates, of which the scoring script kept 18, killed 5, and returned a rerun for 3 whose data could not be trusted or whose tasks could show no gain.

How text reaches the agent matters more than who wrote it: a rule is loaded by the harness whenever the agent reads a matching file, while a skill enters the context only if the agent chooses to open it. Of the 18 items kept, 16 are rules; the two skills are both path-gated to a specific directory. A path-gated candidate was listed to the agent in 66 attempts and opened in 28, while an always-on candidate was listed in 198 attempts and opened in none.

The loop targets a single developer's own repository: tasks come from that developer's sessions, the baseline is the harness currently in use, and it is re-measured in the same sweep as the change. The evaluation runs on a synthetic repository built for the purpose, with four house rules a machine can check, a scripted developer that replies to each mistake in the same words, and Claude Haiku 4.5 in every role; the four evaluation runs each start from a fresh copy of the repository. The source proposes next steps: a real repository whose conventions the model does break, and a sandbox that holds a single commit with the harness installed as untracked files, which would close the route by which rollouts read earlier fixes from git history.

The source states that the gate test shows only that text which never fires is never kept, not that the gates catch harm: the harmful skills were never opened, and the harmful rules, which did load, were screened on tasks the live harness had failed, where no gain could appear, so the scoring script reached no verdict. Only 2 of 4 known-good candidates were kept, the two always-on skills being lost because the agent never opened them. Under the stronger Claude Sonnet 5 the R1 harness gained only 5.6 points, with an interval of [−2.8, +13.9], and one prune pass flagged 2 of its 4 items as removable, though a single pass cannot tell a small effect from none. On structlog the agent was rarely corrected, nothing recurred, the loop kept nothing, and the two harnesses were identical, leaving nothing to compare. The optional judge, replayed afterwards, did not separate kept from killed candidates, on a comparison with only 2 regression kills.