A causal answer to whether an AI agent should be reviewed mid-task: randomized interventions inside one trajectory show effects swing by action and state
Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.