Skip to main content
Back to timeline
arXivSource publication:

Long-running agents quietly take shortcuts, so the workflow state moves out of context: Haiku 4.5 follows the procedure on 139 of 200 accounts

Synopsis

Language-model agents drift from a procedure stated once at the start of a long unattended run; rewriting that procedure as an executable finite-state machine enforced by the runtime at tool calls, state transitions and turn endings let Claude Haiku 4.5 average 139.2 faithful and correct decisions on a 200-account SOP-Bench batch, against 48.4 for the strongest baseline, Remind-loop. Restating the instructions or resuming a stopped agent only moves the drift to another shortcut, and only closing all three routes held the procedure to the last account.

Source-provided article image: Keeping LLM Agents on Procedure: Executable Finite-State Control for Long-Horizon Instruction Fidelity
arXiv

Interpretation

Agents leave a procedure during long batches not because they lack knowledge of it but because the runtime leaves shortcuts open; holding the workflow state outside the model's context and enforcing it keeps the procedure through the whole batch. Putting the procedure in the first message, in the system prompt or in a per-account reminder, or resuming the agent after it stops, only moved the drift from one exit to another. On the 200-account SOP-Bench batch, Claude Haiku 4.5 with the finite-state machine averaged 139.2 faithful and correct decisions against 48.4 for the strongest baseline, Remind-loop, positive in 7 of 8 seeds with a mean difference of 90.9 and a 95% confidence interval of [46.2, 130.2]; every window of 50 accounts stayed between 31.9 and 37.6, while no baseline window after the first 50 exceeded 12.

The control writes the procedure as a finite-state machine whose state is kept by a daemon outside the model's context, so context compaction cannot erase it and a fresh context can read it back. The machine intercepts the agent loop at three points: a tool-call gate admits only the tools the current state permits, a transition guard runs checks before the state may advance, and a stop gate refuses to end the turn before the workflow is complete. The main experiment's guards read only which tools were called, for which account and in what order, never data values or correct answers; ablations show the stop gate alone made the agent decide without looking at the data, the gates alone kept compliance at 93.0% but the agent tried to end its turn 44.7 times per session, and both together gave 128.0 faithful and correct decisions.

Perspective

The demonstrated scope is the referral-abuse-detection task of SOP-Bench, the Claude Code runtime, and three models, Claude Haiku 4.5, Claude Sonnet 4.6 and Qwen3.8-flash; the analyses extend to a debugging procedure on SWE-bench Verified, a customer-service benchmark, and AutoResearch, month-end close and a DuckDB optimizer. The authors call for evaluation over more procedures, runtimes and agent architectures, and for methods that automatically generate and verify workflow specifications.

The pre-registered P3 prediction failed on its compliance condition: in 3 of 8 seeds the agent investigated several accounts and then submitted their decisions together from one state, the route left open because static permissions cannot refer to the current account. Sonnet 4.6 has only 3 seeds, its comparison with Remind-loop triggered the pre-registered falsification rule, and the authors call those comparisons descriptive. In the customer-service benchmark the controller itself emits 62% of the agent turns, so the result measures the whole controller, and the first frozen Retail workflow lost because its checks were too strict. Enforcement constrains how the agent works, not the quality of its decisions.