Robustness tests for tool-using agents now check whether the task survived the edit: rescoring one trajectory set drops Mistral's attacked tasks from 64 to 26

Robustness evaluation for tool-using agents now decides whether an edited test case still tests the original task, instead of judging only what the agent does in the changed environment. The prior practice accepted any well-formed edit to an injection field, or bounded perturbations with an embedding-distance threshold, and neither checked the facts that define the task. On 64 test tasks and 14,336 stored trajectories, embedding thresholds across four encoders allowed 75% to 100% of task-changing substitutions, such as swapping the payee, yet rejected 24% to 45% of valid hostile notes.

A contract splits the task definition into protected facts, protected relations, an authorized outcome, and feasibility, and a candidate must pass all four checks to count as a valid test. Reference replay had served as feasibility evidence, but a failed replay shows only that one saved plan no longer works, and a successful replay does not show the task is unchanged. Across 96 candidates in 16 finite-state tasks, replay succeeded on only 32 of the 64 valid candidates; when the primary route was disabled or renamed the backup route still reached the original outcome, while all 16 expanded-outcome candidates replayed successfully yet changed what counts as success.

What a threshold admits changes robustness conclusions, because every counted attacker success can come from a task-changing substitution. At the task level, the threshold policy counted a successful attack on all 64 of Mistral's tasks. Under the same threshold the contract policy counted 26, the 38 dropped tasks had every attacker success supplied by task-changing substitutions, a fall of 59.4 percentage points with a paired bootstrap 95% interval, and the threshold covered only 273 of 640 valid hostile candidates, so removing it raised Mistral's count from 26 to 48.

Contracts target structured task specifications that name protected facts, protected relations, allowed edits, and an authorized outcome, which the controlled corpus, the finite-state diagnostics, and the Banking, Workspace, and Slack domains of AgentDojo all supply. The intended reader is a researcher or engineer designing or reviewing agent robustness evaluations, and the setting is screening out task-changing candidates before the agent runs. The material lists a finite, non-adaptive attack library and the absence of contract enforcement over long-lived memory or multi-agent interaction as natural next steps.

The controlled corpus and finite diagnostics use deliberately structured cases, so they show that the tested policies can fail and how validity decisions affect scores, but they do not estimate how often these cases occur in practice. The specification counts reflect three specifications that were written, not disagreement among evaluators. The evidence covers four local 7-8B models and selected AgentDojo tasks, so it gives no general rates for frontier agents or other benchmarks, and the bootstrap intervals resample test tasks rather than measuring deployment uncertainty. Contracts do not discover the correct task definition and do not establish semantic equivalence or agreement with every human judge.