A causal answer to whether an AI agent should be reviewed mid-task: randomized interventions inside one trajectory show effects swing by action and state
Synopsis
Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.
Interpretation
Agent evaluation moves from task success alone to estimating the causal effect of one mid-trajectory review on the final outcome. Benchmarks fix tasks, environments and scoring criteria, so they characterize capability and safety but cannot say whether intervening at a given step during deployment would change the result. In τ-bench Retail, review effects for the GPT-5 nano agent are estimated positive for both read and write calls; at tool level the confidence intervals lie above zero for order cancellation and item changes and include zero for payment changes.
The design flips an independent coin at every eligible decision in each trajectory to choose review or direct execution, then lets the agent keep interacting with the environment. Micro-randomized trials were developed for settings such as mobile health, repeatedly randomizing interventions as context evolves; here the same design is applied to agent tool calls and code submissions. τ-bench Retail uses 114 original tasks plus 114 variants with ten randomizations per agent; ToolSandbox collects 3,096 trajectories per agent; ImpossibleBench analyzes 1,218 valid episodes.
The same randomized trajectories also evaluate selective policies that review only under certain histories, compared with never reviewing. Earlier trajectory-level randomized trials assigned treatments at the task or whole-trajectory level, so they could not evaluate history-dependent intervention policies. For nine selective review policies in τ-bench Retail, both the structural nested mean model estimates and direct Monte Carlo evaluation with 456 rollout episodes per policy are positive, and the 95% confidence intervals overlap for all nine policies.
Perspective
Experiments cover the existing benchmarks τ-bench Retail, ToolSandbox and ImpossibleBench, with agents GPT-5 nano, Claude Haiku 5.5 and Qwen3.5 Flash and GPT-6 Luna as reviewer; conclusions are limited to the intervention effects estimated under these settings.
Structural nested mean model estimates assume conditional blip effects belong to the specified parametric class, and confidence intervals cover the target causal effects only when that model is well specified; reported intervals are pointwise without multiple-comparison correction; the authors also note the experiments use relatively small models and benchmarks that frontier models might saturate.